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ABSTRACT 



In 1994 the General Assembly of the International 
Association for the Evaluation of Educational Achievement (IEA) decided to 
undertake a study on civic education. This volume reports on Phase 2 of the 
project, which consisted of a test (keyed cognitive items) and a survey 
(un-keyed attitudinal and behavioral items) administered in each 
participating country to representative samples of about 3,000 students in 
the modal grade for 14 -year-olds . A questionnaire was administered to 
civic-related teachers and to school principals. This report will give 
educators, policymakers, and the public information about what 14 -year-old 
students know and think about democratic institutions and processes. It also 
provides a snapshot of the civic activities of young people and an indication 
of their intentions regarding future participation in civic affairs. It is 
divided into the following chapters: (1) "Introduction to the IEA Civic 

Education Study"; (2) "Instrument Development, Sampling, Testing and Quality 
Control"; (3) "Knowledge of Content and Skills in Interpreting Civic 
Information"; (4) "Students' Concepts of Democracy, Citizenship and 
Government"; (5) "Students' Attitudes toward the Nation, the Government, 
Immigrants and Women's Political Rights"; (6) "Students' Civic Engagement and 
Political Activities"; (7) "Students' Views' of Opportunities for Civic 
Engagement in Classrooms, Schools and Youth Organizations"; (8) "A Model for 
Explaining Students' Civic Knowledge and Engagement"; (9) "The Teaching of 
Civic Education"; and (10) "Civic Knowledge and Engagement: A Synthesis." 
Appendices contain examples of items from the civic knowledge test; 
item-by-score maps for scales reported in chapters 5 through 7; classical 
psychometric indices (selected) ; standard deviations of total civic 
knowledge; and teachers' reports. (Contains extensive references.) (BT) 
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The International Association for the Evaluation of Educational 
Achievement, known as IEA, is an independent, international consortium 
of national research institutions and governmental research agencies, with 
headquarters in Amsterdam. Its primary purpose is to conduct large-scale 
comparative studies of educational achievement with the aim of gaining 
more in-depth understanding of the effects of policies and practices 
within and across systems of education. 
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In 1994 the General Assembly of IEA decided to undertake a study on civic 
education. It was not the first time that IEA had focused on this issue. As early 
as 1971 it had taken its first look at civic education, in the context of the so- 
called Six Subject Study. But the decision, in 1994, to look again at the 
subject was a sound one given the huge changes by then facing many 
countries as a result of the events of the late 1980s and early 1990s. The 
considerable task of establishing or re-establishing democratic governments in 
a number of countries highlighted even more the need to develop citizenship 
and the role that educational systems could play in meeting that aim. Assessing 
civic education was important not only for those countries, however, but also 
for societies with long-established democratic traditions. In general, it could be 
said that changes in the political, social and educational scenes of many 
countries suggested the timeliness of this new study, particularly in terms of its 
potential to make a substantial contribution to an understanding of these 
changes. 

The International Association for the Evaluation of Educational Achievement 
(IEA) was, and is, in an excellent position to make such contribution. It was 
founded in 1959 for the purpose of conducting comparative studies focusing 
on educational policies and practices in various countries and educational 
systems around the world. Since that time, it has completed a significant 
number of studies in different subjects, as varied as reading literacy, 
mathematics, science, pre-primary education, and information and 
communication technologies in education or languages, among others. With its 
nearly 60 member countries, its Secretariat located in Amsterdam and a 
number of interconnected research centres in all continents, IEA is in a very 
sound situation to produce cross-country comparative studies that are based on 
rigorously collected and analysed data. 

In 1994, the IEA General Assembly approved the Civic Education Study as a 
two-phased project. The aim of Phase 1 was to collect extensive information 
describing the circumstances, content and process of civic education in 
participating countries. In doing this, IEA summarised what country experts 
considered 14-year-old students should know about a number of topics 
related to democratic institutions and citizenship, including elections, 
individual rights, national identity, political participation and respect for ethnic 
and political diversity. 

The results of Phase 1 were presented in Civic education across countries: Twenty- 
four national case studies from the IEA Civic Education Project, a book that received 
wide recognition among researchers, practitioners and policy-makers. Its 24 
national case studies were written mostly by National Research Coordinators, 
and also took into account opinions expressed by National Expert Panels. 



The information collected in Phase 1 was also used for preparing Phase 2. 

This second part of the project consisted of a test (keyed cognitive items) and a 
survey (un-keyed attitudinal and behavioural items) administered in each 
participating country to representative samples of about 3,000 students in the 
modal grade for 14-year-olds. A questionnaire was also administered to civic- 
related teachers and to school principals. Data were collected in spring 1999 
in most of the participating countries. 

This current publication, Citizenship and education in twenty-eight countries, 
presents the first results of Phase 2 of the study. It follows a style similar to 
that traditionally used by IEA, and it complements the more qualitative 
approach of the first volume by reporting quantitative information from the 
tests, surveys and questionnaires. Together, the two publications provide a 
complete and remarkable picture of civic education policies, practices and 
results across countries in the late 1990s. 

Having identified and discussed the outcomes of our respective countries in an 
international context, we know that the time has arrived to pay special 
attention to the factors that merit consideration and possible action. Wise 
action requires a deep knowledge of the field. The comparative view helps us 
set our reflections in a context that allows us to interpret and to explain. In 
this manner, the value of an international approach can be truly realised. It is 
this realisation that is exactly the kind of contribution IEA can make to the 
development of education and educational systems. In the end, our activities 
can only be justified if they contribute to the advancement of societies made 
up of better-developed individuals. 

IEA is particularly grateful to the following organisations, which are the major 
contributors to the international overhead of Phase 2 of the study: the 
Deutsche Forschungsgemeinschaft (DFG or German Science Association) and 
the William T. Grant Foundation of New York. As in all IEA studies, 
individual participating countries also provided funding. 

Citizenship and education in twenty-eight countries represents the results of 
initiatives developed by the International Steering Committee of the study and 
by the National Research Coordinators and National Experts. Special thanks 
should necessarily go to the International Steering Committee and the 
International Coordinators, Professor Judith Torney-Purta (University of 
Maryland) and Professor Rainer Lehmann (Humboldt Universitat zu Berlin). 

As the leaders of this study, they have provided its special spirit, and so 
deserve our recognition and thanks. 



Alejandro Tiana 

CHAIR OF IEA 
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CHAPTER 1 









During a single decade, beginning in the late 1980s, initiatives toward 
democratic reform took place across the world. New constitutional regimes 
came into being. In countries that were establishing or re-establishing 
democracies after a period of non-democratic rule, the general public as well as 
leaders realized that major changes in formal and informal civic education 
were required to prepare young people for this new social, political and 
economic order. What those changes should be and how they should be 
initiated was not clear, however. 

During the same period, many well-established democracies recognized that 
their own methods of preparing young people for citizenship were far from 
ideal. In some countries, young adults were unlikely to vote or participate in 
other conventional political activities. Youth demonstrated gaps both in their 
understanding of the pivotal ideas of democracy and in their knowledge of 
existing political structures. Few seemed to have the skills to analyze political 
issues presented in the newspaper or on television news (if they paid attention 
to these media at all). In some countries, ‘civil society’, the web of community 
groups and private associations that operates independently from government 
and market sectors, seemed to be drawing in few youth. 

These issues called for a rethinking of civic education, a challenge that many 
countries began to face during the 1990s. The home, school, community, peer 
group and mass media remained important considerations, but there were also 
new factors. A global youth culture was intensifying in its importance and 
nurturing common aspirations for freedom along with shared consumer tastes. 
Environmental organizations and human rights groups often involved youth 
on an equal footing with adults and seemed poised to replace more 
hierarchically organized political groups such as political parties. An enhanced 
emphasis on individual choice challenged long-standing views of youth as 
passive recipients of lessons from their elders. Young people could be seen as 
active constructors of their own ideas, as people whose everyday experiences 
in their homes, schools and communities influenced their sense of citizenship. 

In light of these factors, questions were asked regarding the direction that 
should be taken in order to enhance the contribution of schools to citizenship. 
Should the emphasis be on teaching factual information about the country and 
its structure of government? Should it be instead on making young people 
aware of political issues or interested in news provided by the mass media? 
Should they be encouraged to join explicitly political organizations, such as 
parties? Or should the emphasis be on providing opportunities for 
involvement in environmental organizations, or groups providing assistance to 
the community, or school councils? And how could community support be 
gained for programs that would provide more rigorous study of citizenship 
within schools and more opportunities for the practice of civic education 
outside schools? These questions were faced by countries where schools 
offered courses labeled civic education as well as by countries where civic- 
relevant material was embedded in history courses or spread throughout the 
curriculum. 
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No single piece of research could be expected to fully answer questions such 
as these. However, it was clear that rigorous cross-national research in civic 
education could play a role in providing an empirical foundation for policy- 
makers, those who design curricula and those who prepare educators, as well 
as for teachers or youth workers and the public. 

Educational policy-makers in this area often operate with many aspirations but 
little up-to-date information about civic knowledge, attitudes and behavior in 
their own countries. On a cross-national basis, where the experience of other 
countries might provide a rich set of possibilities and comparisons, data were 
even more limited. The International Association for the Evaluation of 
Educational Achievement (IEA) Civic Education Study was designed to address 
this gap and to create the possibility of a rigorous data-based approach to a 
number of questions with implications for policy and educational practice. IEA 
has served as a coordinating organization for comparative research in various 
school subject areas since the 1960s. The best-known IEA study is TIMSS (the 
Third International Mathematics and Science Study), but over the years, other 
areas, including reading literacy and civic education, have been surveyed. 

Specifically, what can a cross-national study contribute to the educational 
debate? It can document similarities and differences in student outcomes, and 
also in the organization and content of programs and practices across the 
world. Another contribution of well-designed cross-national research is that it 
can show connections between practices or policies and the achievement of 
certain goals for civic education in different nations. It can also foster 
awareness of the importance of education for citizenship in its many forms. 

The goal of the IEA Civic Education Study is to identify and examine in a 
comparative framework the ways in which young people are prepared to 
undertake their role as citizens in democracies. One focus of the study is the 
school. This is not limited to the formal curriculum in any particular school 
course, but includes several subject areas across the curriculum. Opportunities 
for discussion in the classroom and participation in the school are important, 
as are textbooks and curriculum. A second focus is on opportunities for civic 
participation outside the school, especially in the community. 

A primary purpose is to obtain a picture of how young people are initiated 
into the political communities of which they are members, including in- and 
out-of-school experience. The study concentrates on political processes and 
institutions. But the concept ‘political’ is used in a fairly broad sense and is not 
limited to formal political organizations or legislative structures. 

The remainder of this chapter sets the IEA Civic Education Study within 
several frameworks: 

1 . the history and structure of IEA (the sponsoring organization) and the 
participating countries; 

2. the context of its two-phased design; 

3. the context of existing theoretical and research frameworks; and 

4. the structure of a set of policy questions. 
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THE STUDY WITHIN THE CONTEXT OF IEA A^O THE 
PARTICIPATING COUNTRIES 
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The Organization of the Study by IEA 

Responding to the expressed need of many countries for empirical data as they 
began to rethink their civic education programs in the early 1990s, the 
International Association for the Evaluation of Educational Achievement 
decided to mount a cross-national study of civic education. A two-phase study 
was designed. The first phase would consist of qualitative case studies that 
would examine the contexts and meaning of civic education in different 
countries and provide background for the development of the instruments to 
be administered to students and teachers. The second phase would consist of a 
test of civic knowledge and a survey of civic engagement for statistical 
analysis. It was expected that the project would complete its testing of the 
‘standard population’ of 14-year-olds before the end of the 20th century and 
release an international report early in the 21st. It was also expected that the 
testing of an older population would be completed in 2000 in a smaller 
number of countries and that the findings would be reported approximately 
one year after the report on the standard population. 

In 1971, the IEA conducted a civic education survey that employed nationally 
representative samples of three age groups in the Federal Republic of Germany, 
Finland, Ireland, Israel, Italy, the Netherlands, New Zealand, Sweden and the 
United States (Torney, Oppenheim & Farnen, 1975). About 30,000 students 
responded to instruments measuring knowledge and attitudes, while 5,000 
teachers and 1,300 principals and headmasters described pedagogy and the 
characteristics of schools. The instrument included a test of civic knowledge, 
measures of support for democratic values (including tolerance and support for 
women’s political rights), support for the national and local government, and 
participation in political activities. No country’s 14-year-olds achieved high 
scores on all of these factors. There were substantial gender differences, with 
males scoring higher on civic knowledge and on participation in political 
discussion, and females scoring higher on support for democratic values. 
Another major finding from this study was that stress on rote learning and on 
patriotic ritual in the classroom tended to be negatively related to civic 
knowledge and democratic attitudes, while the opportunity to express an 
opinion in class had a positive impact. The socioeconomic status of the family 
and the type of school were statistically controlled in these analyses, and the 
predictors of success were similar within each of the nine countries. As 
interesting as these findings were, the intervening 20 years had seen many 
changes in schools and political systems, thereby raising new issues and 
intensifying concern about old ones. 

In 1994, the governing body of IEA, its General Assembly, voted to undertake 
the current Civic Education Study because of interest among its diverse 
member countries, many of which were experiencing political, economic and 
social transitions. An International Steering Committee to guide the research 
and an International Coordinating Center to coordinate its day-to-day 
operations were appointed. The international oversight and coordination of 
this study has been funded by agencies and foundations in Germany and the 
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United States, by the IEA organization, and by contributions from 
participating countries. The two senior authors of the current volume have 
been, respectively, the Chair of the International Steering Committee (at the 
University of Maryland in College Park, USA) and the International 
Coordinator (at the Humboldt University of Berlin, Germany). National 
Research Coordinators were appointed in each participating country. Their 
work, including data collection, has been funded by governments and 
foundations within each country. 

Participating Countries 

Twenty-eight countries accepted IEA’s invitation, sent to all 5 1 member- 
countries, to participate in the test and survey reported in this volume. (Figure 
1.1 lists the 28 countries.) Approximately two-thirds of the participating 
countries collaborated in the research from the beginning. They: 

• completed case studies for Phase 1 (thus influencing the framework and 
item development); 

• sent representatives to National Research Coordinators’ meetings beginning 
in 1995; 

• contributed items or critiqued instruments as they were being developed; 
and 

• pre-piloted and pilot-tested the preliminary forms of the test and survey and 
examined the results. 

The other one-third of the countries joined the study later; the last in 
November 1998. 

The study was a massive one both in the breadth of its coverage relating to the 
material identified in Phase 1 and in the number of respondents (nearly 
90,000). 

Three aspects of the participating countries are important in terms of 
understanding the data collected: national demographics, characteristics of the 
educational system, and characteristics of the political system. 

Table 1 . 1 presents selected demographic data from the participating countries. 
Both large and small countries participated in the study. On the United 
Nations Human Development Index, about three-quarters of the countries fall 
into the highly developed category and about one-quarter into the medium 
developed category. Population, GNP per capita and unemployment rates are 
also found in the table. 

Table 1.2 presents some educational characteristics of participating countries. 
Adult literacy levels are generally high in participating countries. The table 
also shows that there is a great deal of variation in the number of Internet 
hosts per country (although these figures are changing rapidly), and it 
provides information about expenditures for public education. 

Table 1.3 presents political characteristics of participating countries. These 
include the number of political parties represented in the lower house (ranging 
from two to 1 1), voter turnout at the last election for the lower house (ranging 
from 36 to 95 percent), and percentage of seats in the national legislature held 
by women (ranging from 5.6 to 42.7 percent). All participating countries can 



CHAPTER 1 INTRODUCTION 



17 



Figure 1.1 Countries Participating in the IEA Civic Education (CivEd) Study 




Participating Countries: 

• Australia 

• Belgium (French)* 

• Bulgaria 

• Chile 

• Colombia 

• Cyprus 

• Czech Republic 

• Denmark 

• England 

• Estonia 



• Finland 

• Germany 

• Greece 

• Hong Kong (SAR)** 

• Hungary 

• Italy 

• Latvia 

• Lithuania 

• Norway 



• Poland 

• Portugal 

• Romania 

• Russian Federation 

• Slovak Republic 

• Slovenia 

• Sweden 

• Switzerland 

• United States 



* Only the French educational system in Belgium participated. 
** Special Administrative Region of China. 



be classified as liberal or electoral democracies, according to Diamond (1999). 
The age at which people can cast their first vote is 18 in all the countries in 
the study. 

THE TWO PHASES OF THE IEA CIVIC 'EDUCATION STUDY 

When IEA first discussed undertaking a study in this area, relatively little was 
known about what civic education meant in many countries. For this reason, as 
already mentioned, the study was designed to begin with a more qualitative 
case study phase and to follow it with a second phase including a test and 
survey more typical of IEA studies. 

In Phase 1, each participating country completed a national case study of civic 
education, submitting four documents to an international document base: 

1. A plan for Phase 1, including a summary of the current status of civic 
education. 

2. A review of empirical literature concerning civic education and the social 
and political attitudes and behavior of youth. 

3. Information regarding current policies, practices and issues concerning 
preparation for citizenship organized around a set of 1 8 case study 
framing questions. 
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Table 1.1 Selected Demographic Characteristics of Participating Countries 



Country 


Population 
(in millions) 

(1998) 


Human Development 
Index b 

(value, rank & category) 
(1998) 


GNP per capita 
(in US $) c 

(1998) 


Unemployment Rate 
(% of labor force) 

(1998) 


Australia 


18.5 


0.93 (4) High 


20,640 


7.6 d 


Belgium (French) 9 


10.1 


0.93 (7) High 


25,380 


8.8 


Bulgaria 


8.3 


0.77 (60) Medium 


1,220 


12. 2 e 


Chile 


14.8 


0.83 (38) High 


4,990 


N/A 


Colombia 


40.8 


0.76 (68) Medium 


2,470 


N/A 


Cyprus 


0.8 


0.89 (22) High 


11,920 


N/A 


Czech Republic 


10.3 


0.84 (34) High 


5,150 


6.5 


Denmark 


5.3 


0.91 (15) High 


33,040 


5.1 


England 9 


58.6 


0.92 (10) High 


21,410 


6.3 


Estonia 


1.4 


0.80 (46) High 


3,360 


5.1 e 


Finland 


5.2 


0.92 (11) High 


24,280 


11.4 


Germany 


82.1 


0.91 (14) High 


26,570 


9.4 


Greece 


10.6 


0.88 (25) High 


11,740 


9.6 f 


Hong Kong (SAR) 


6.7 


0.87 (26) High 


23,660 


N/A 


Hungary 


10.1 


0.82 (43) High 


4,510 


8.0 


Italy 


57.4 


0.90(19) High 


20,090 


12.2 


Latvia 


2.4 


0.77 (63) Medium 


2,420 


9.2 e 


Lithuania 


3.7 


0.79 (52) Medium 


2,540 


6.9 e 


Norway 


4.4 


0.93 (2) High 


34,310 


3.3 


Poland 


38.7 


0.81 (44) High 


3,910 


10.6 


Portugal 


9.9 


0.86 (28) High 


10,670 


4.9 


Romania 


22.5 


0.77 (64) Medium 


1,360 


10.3 e 


Russian Federation 


147.4 


0.77 (62) Medium 


2,260 


13. 3 e 


Slovak Republic 


5.4 


0.83 (40) High 


3,700 


15. 6 e 


Slovenia 


2.0 


0.86 (29) High 


9,780 


14. 6 e 


Sweden 


8.9 


0.93 (6) High 


25,580 


8.2 


Switzerland 


7.3 


0.92 (13) High 


39,980 


4.2 f 


United States 


274.0 


0.93 (3) High 


29,240 


4.5 


a Figures for all of Belgium used for Belgium (French); figures for United Kingdom used for England. 


b The Human Development Index (HDD is a composite index that reflects three basic dimensions: (a) longevity (life 


expectancy at birth); (b) knowledge (adult literacy and combined gross primary, secondary and tertiary enrollment 


ratio); and (c) standard of living (adjusted per capita income in PPP US$).The HDI value ranges from 0 to 1. 


Countries are divided into categories of high, medium and low human development, and are ranked, 
c Data refer to GNP calculated using the World Bank Atlas method, in current US dollars. 


d Data refer to 1998-99. Source: W. McLennan, Year book Australia, No. 82, p. 123, Canberra, Australian Bureau 
of Statistics. 


e Data are estimates by the UN Economic Commission for Europe, based on national statistics. They refer to registered 
unemployment, which is likely to bias unemployment figures downward, 
f Data refer to 1997. 



Sources: 

All column sources are from the Human development report 2000, Oxford/New York, Oxford University Press (published for the United 
Nations Development Programme), unless noted otherwise. 

Population (pp. 223-26). 

Human Development Index (pp. 157-60). 

Gross National Product per capita (G N P) (pp. 202-04). 

Unemployment rate (pp. 241-42). 
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Table 1.2 Selected Educational Characteristics of Participating Countries 



Country 


Adult Literacy Rate 

(in %) 

(1998) 


Public Education 
Expenditure 

(as % of GNP) C 
(1995-1997) 


Internet Hosts 
(per 1000 people) 

(1998) 


Australia 


99.0 b 


4.4 d 


40.1 


Belgium (French) 3 


99.0 b 


3.2 e 


20.6 


Bulgaria 


98.2 


3.2 


1.2 


Chile 


95.4 


3.3 


2.0 


Colombia 


91.2 


4.4 ' 


0.4 


Cyprus 


96.6 


4.5 


7.9 


Czech Republic 


99.0 b 


5.1 


8.4 


Denmark 


99.0 b 


8.1 


56.3 


England 3 


99.0 b 


5.3 


24.6 


Estonia 


99.0 b 


7.2 


16.6 


Finland 


99.0 b 


7.5 


89.2 


Germany 


99.0 b 


4.8 


17.7 


Greece 


96.9 


3.1 


4.7 


Hong Kong (SAR) 


92.9 


2.9 


12.4 


Hungary 


99.3 


4.6 


9.4 


Italy 


98.3 


4.9 


6.7 


Latvia 


99.8 


6.3 


5.8 


Lithuania 


99.5 


5.5 


2.7 


Norway 


99.0 b 


7.4 


71.8 


Poland 


99.7 


4.6 9 


3.4 


Portugal 


91.4 


5.8 


5.6 


Romania 


97.9 


3.6 


1.1 


Russian Federation 


99.5 


3.5 


1.2 


Slovak Republic 


99.0 b 


5.0 


4.1 


Slovenia 


99.6 


5.7 


11.5 


Sweden 


99.0 b 


8.3 


42.9 


Switzerland 


99.0 b 


5.4 


34.5 


United States 


99.0 b 


5.4 e 


112.8 


a Figures for all of Belgium used for Belgium (French); figures for United Kingdom used for England, 
b Human Development Report Office estimate. 

c Data refer to the most recent year available during the period 1995-97. 


d Source: W. McLennan, Year book Australia, No. 82, p.285, Canberra, Australian Bureau of Statistics. 


e Data refer to a year other than those encompassed by 1995-97. Belgian data are from Human development report 
1 999 and refer to years 1993-96. 
f Data refer to expenditures by Ministry of Education only. 


g Source: K. Konarzewski (2000) Educational infrastructure in the first year of educational system reform in Poland, 
Poland, Institute for Public Issues. 



Source: 

All column sources are from the Human development report 2000, Oxford/ New York, Oxford University Press (published for the United 
Nations Development Programme), unless noted otherwise. 

Literacy rate (pp. 157-60). 

Public education expenditures (pp. 194-97). 

Internet hosts (pp. 194-97). 
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Table 1.3 Selected Political Characteristics of Participating Countries 



Country 


Seats in Parliament Held 
by Women as of 
February 2000 

(% of total) 


Voter Turn-out at Latest 
Elections' 1 

(%) 


Political Parties 
Represented in Lower 
or Single House 


Australia 


25.1 


95 


5 C 


Belgium (French) 3 


24.9 


91 


11 


Bulgaria 


10.8 


68 


5 


Chile 


8.9 


86 


7 d 


Colombia 


12.2 


45 


2 d 


Cyprus 


7.1 


93 


5 


Czech Republic 


13.9 


74 


5 


Denmark 


37.4 


86 


10 


England 3 


17.1 


72 


10 d 


Estonia 


17.8 


57 


7 


Finland 


36.5 


65 


7 d 


Germany 


33.6 


82 


5 


Greece 


6.3 


76 


5 


Hong Kong (SAR) 


N/A 


N/A 


N/A 


Hungary 


8.3 


56 


6 d 


Italy 


10.0 


83 


9 e 


Latvia 


17.0 


72 


6 


Lithuania 


17.5 


53 


6 d 


Norway 


36.4 


78 


7 d 


Poland 


12.7 


48 


6 


Portugal 


18.7 


62 


5 


Romania 


5.6 


76 


7 


Russian Federation 


5.7 


62 


7 d 


Slovak Republic 


14.0 


84 


6 


Slovenia 


10.0 


74 


8 


Sweden 


42.7 


81 


7 


Switzerland 


22.4 


43 


8 d 


United States 


12.5 


36 


2 d 


a Figures for all of Belgium used for Belgium (French); figures for United Kingdom used for England, 
b Voter turn-out for lower or single house. 


c Source: W McLennan, Year book Australia, No. 82, Canberra, Australian Bureau of Statistics. 


d There are also independent and other parties not sufficiently represented to constitute a parliamentary group, 
e Source: Italian Parliament web site. 


The age of first vote for all countries is 18 years. (Source: The International Institute for Democracy and Electoral 
Assistance (IDEA) web site. Http://www.idea.int/turnout/>) 



Source: 

All column sources are from the Human development report 2000, Oxford/New York, Oxford University Press (published for the United 
Nations Development Programme), unless noted otherwise. 

Seats in parliament held by women (pp. 165-68). 

Voter turn-out (pp. 243-46) 

Political parties (pp. 243-46). 
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4. An in-depth analysis of core issues in democracy, citizenship, national 
identity and diversity, including an examination of textbook treatment of 
these issues and teaching methods. 

Many countries collected data from focus groups or interviews in addition to 
examining printed materials as they prepared these documents. All of this 
material provided a view of the participating countries’ intended curricula in 
civic education as well as extensive contextual material. 

Each National Research Coordinator also prepared a chapter for Civic education 
across countries: Twenty-four national case studies from the IEA Civic Education Project 
(Torney-Purta, Schwille & Amadeo, 1999), the first volume arising out of 
Phase 1 of the study. The documents used as the basis for this publication have 
also been used in the preparation of the second volume from Phase 1 (Steiner- 
Khamsi, Torney-Purta & Schwille, forthcoming), which reports the findings of 
cross-national analysis of the case study material. The themes identified during 
the first phase are reviewed under the section on the policy questions in this 
chapter and in the next chapter. Chapter 2 also describes the development of 
the test and survey used in the second phase. 

This second phase of the study, reported in this present volume, tested and 
surveyed nationally representative samples of 14-year-olds in 28 countries 
regarding their knowledge of civic-related content, their skills in 
understanding political communication, their concepts of and attitudes toward 
civics, and their participation or practices in this area. The instrument drew 
from material submitted during Phase 1 and benefited from the input of 
members of the International Steering Committee, IEA’s Technical Executive 
Group, National Research Coordinators and National Advisory Committees 
throughout the five-year process of framework development, item writing, pre- 
piloting and piloting and final item choice. 

TOED) METICAL FRAMEWORKS GUIDING THE OESIGM 

The National Research Coordinators at their first meeting took on the task of 
developing an overall model for the study. This model, described as the 
Octagon, graphically represents a framework for organizing the information 
being collected in both phases (Figure 1.2). It is a visualization of ways in 
which the everyday lives of young people in homes, with peers and at school 
serve as a ‘nested’ context for young people’s thinking and action in the social 
and political environment. Learning about citizenship involves engagement in 
a community and development of an identity within that group. These 
‘communities of discourse and practice’ provide the situation in which young 
people develop progressively more complex concepts and ways of behaving. 
The model has its roots in two contemporary psychological theories — 
ecological development (Bronfenbrenner, 1988) and situated cognition (Lave 
& Wenger, 1991; Wenger, 1998). At the center of this model is the individual 
student. The public discourse and practices of the society have an impact on 
the student through contacts with family (parents, siblings and sometimes 
extended family), school (teachers, implemented curriculum and participation 
opportunities), peer group (both in and out of class), and neighbors (including 
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Figure 1.2 Model for IEA Civic Education Study 




people in out-of-school youth organizations). Earlier work in political 
socialization usually referred to these groups of people as ‘agents’ of 
socialization. 

In addition to these face-to-face relationships, there is also a broader society 
that has an impact through its institutions and the mass media. The outer 
octagon in Figure 1.2, which circumscribes these processes, includes 
institutions, processes and values in domains such as politics, economics, 
education and religion. It also includes the country’s position internationally, 
the symbols or narratives important at the national or local level, and the 
social stratification system, including ethnic and gender-group opportunities. 

Other models have also influenced the study. Sociologists and political 
scientists see the IEA study in relation to studies of political socialization — a 
sub-field of political science research that was popular 20 to 25 years ago and 
seems currently to be experiencing renewed interest (Niemi & Hepburn, 1995; 
Flanagan & Sherrod, 1998). Social scientists link studies in this area to recent 
surveys of adults concerned with social capital (Van Deth, Maraffi, Newton & 
Whiteley, 1999), democratic transitions (Diamond, 1999; Dalton, 2000), post- 
materialist values (Inglehart, 1997; Inglehart & Baker, 2000) and political 
culture and citizenship (Norris, 1999). 

These models from the social sciences suggest that young people move from 
peripheral to central participation in a variety of overlapping communities (at 
the school or neighborhood level, as well as potentially at the national level). 
Learning about citizenship is not limited to teachers explicitly instructing 
young people about their rights and duties. The political community itself (and 
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its everyday practices) surrounds and provides a context for developing 
political understanding (Wenger, 1998; Torney-Purta, Hahn & Amadeo, 
2001 ). 

For young people, the peer group plays a vital role. The reactions of peers to 
ideas and choices are essential parts of the context for civic development. The 
extent to which students are able to incorporate what they are learning into 
meaningful identities is also important. Schools as well as neighborhoods are 
important sites for peer interaction and identity development. 



In addition to these models, a list of policy-relevant questions was developed 
to focus the study and make it useful to those who teach, make education 
policy, educate teachers, prepare curriculum materials, provide guidance to 
student associations and conduct research. The original list of 18 questions has 
been merged into 12 questions. Information from Phase 1 (reported in Torney- 
Purta etal., 1999) and Phase 2 (reported in this volume) is referenced in 
treating each policy question in the following section. 

Some of these policy-relevant questions deal with the organization of educational 
program: 

1 . What is the status of citizenship education as an explicit goal for schools? There is 
considerable diversity among countries in the extent to which the 
preparation of future citizens is thought of as an important responsibility 
for schools. Phase 1 indicated that all the participating countries have 
courses under a variety of titles with specific responsibilities to prepare 
students for citizenship. The aims of civic education are also addressed 
throughout the curriculum and the entire school day, as well as through 
the climate for interaction in the classroom. In many countries, civic 
education courses and programs do not have a high status, however. 
Analysis relating to school experience from Phase 2 is relevant to this 
question (found in Chapters 7 through 9). 

2. To what extent is there agreement among nations about priorities within formal civic 
education? Knowledge of domestic political institutions and traditions is a 
focus in most of the participating countries. Lowering levels of youth 
alienation or raising levels of interest in political participation is also 
important in many. During Phase 1 a high level of unanimity was 
identified across participating countries about the major content domains 
of civic education. These domains encompass democracy and democratic 
institutions, citizenship, national identity, international or regional 
organizations and social cohesion and diversity. Items relating to these 
topics form the core of the Phase 2 test and survey (reported in Chapters 

3 through 9). 

3 . Around what instructional principles and through what courses are formal program 
of civic education organized? There is considerable diversity in the extent to 
which citizenship education is addressed through subjects such as history, 
through more interdisciplinary programs such as social studies or social 
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science, through courses focused on conduct such as moral education, and 
through specific courses in civic education or government. There is also 
variation in the extent to which the community or the school is thought 
of as an arena in which the student should practice citizenship. The case 
studies prepared for Phase 1 showed agreement among specialists that 
civics-related courses should be participative, interactive, related to life in 
school and community, conducted in a non-authoritarian environment, 
cognizant of diversity and co-constructed with parents and the 
community. Many countries, however, saw difficulties in implementing 
this kind of civic education because it is not a curriculum-bound subject. 
Most countries thought that the school had an important role in regard to 
it, however. The Phase 2 results include data from students about their 
opportunities for interactive and participatory experience (especially in 
classroom discussion and in organizations inside and outside the school, 
reported in Chapters 7 and 8, and from teachers about their methods, 
reported in Chapter 9). 

4. To what extent does formal education deal with civic identity development in 
students? In societies that have recently become independent, national 
identity is an especially important component of citizenship. Civic 
education must often balance identities relating to the ideal values of 
democracy with support for the current structure. Phase I of the Civic 
Education Study indicated the complexity of this issue in many countries. 
The data from Phase 2 deal with positive feelings about one’s nation, with 
concepts of the role of the good citizen, and with groups that shape 
identity (reported in Chapters 4 and 5). 

5. To what extent is civic education intended to contribute to the resolution of conflicts 
and tensions between societal groups? Many societies are experiencing such 
tensions. The information collected during Phase 1 indicated that this was 
an area of widespread concern but did not suggest clear-cut directions for 
program development. Some countries experience diversity primarily in 
terms of race or ethnicity; others in terms of immigration (often related to 
diversity in language or religion). Phase 2 assessed attitudes relating to 
support for opportunities for immigrants (reported in Chapter 5). 

Some policy-relevant questions are focused on students: 

6. How do students define and understand the concept of citizenship and related issues? 
Students have developed their own ideas about their political system and 
society, and about what citizenship means within it. The Phase 1 process 
identified major concepts that experts in all the participating countries 
agreed were important. Many country representatives also pointed to 
substantial gaps between the concepts that schools were trying to foster 
and what students actually believed. The Phase 2 data provide descriptive 
information on how students understand citizenship, democracy and 
government. They also allow an analysis of the extent to which 
knowledge of civics relates to expected civic engagement. These data are 
reported in Chapters 3, 4, 6 and 8. 
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7. For what rights and responsibilities of participation are students being prepared in 
their own political system or society? In democratic societies, participation in 
the community and political system is vital, although the nature of that 
participation may vary. Information from Phase 1 indicated that educators 
often seek to make students aware of the excitement of politics and the 
importance of participation. Students, however, often show a general 
disdain for politics. Some countries are responding by using student- 
generated projects, while others are encouraging students to assist others 
in the community. Such programs do not yet exist on a widespread basis 
across countries. The Phase 2 data describe students’ current civic 
participation and their future expectations of participation (reported in 
Chapters 6 through 8). 

8. Do male and female students develop different conceptions of citizenship, and do 
they develop different potential roles in the political process? Beliefs about the 
role of women in politics still vary across countries, even though there 
have been rapid changes in the past decade. Phase 1 indicated that most 
countries did not see gender issues as central in preparation for 
citizenship, although some did refer to the small proportion of women 
holding political office as an issue. Phase 2 data indicate the extent to 
which male and female students see the civic culture and citizenship 
similarly or differently. A set of items relating to support for women’s 
political rights was included in the instrument. These data are reported in 
Chapters 3 through 8. 

9. Are there socioeconomic differences in students' understanding of or attitudes to 
civic-related topics or in the way their civic education is structured? Research in 
political socialization and civic education suggests that there are important 
differences in civic knowledge between students from homes with ample 
educational and economic resources and those from homes that are less 
well endowed. The Phase 1 case studies in a few countries dealt with this 
concern. The Phase 2 analyses presented in this volume address this 
question by looking at the relation of civic education outcomes to a 
measure of home literacy resources (in Chapters 3 and 8). 

Some policy-relevant questions focus on teachers and teaching and on schools: 

10. How do teachers deal with civic education in their teaching, and what is the 
influence of different types of classroom practices? Research suggests that 
different pedagogies make a difference, particularly in terms of whether 
discussion is encouraged and how controversy and conflicting beliefs are 
handled. The Phase 1 material across countries confirmed that teachers are 
expected to balance cognitive, attitudinal and behavioral goals in 
preparing students for citizenship. The relevant Phase 2 data are discussed 
in the chapters where students report about their schools and in the 
chapter about teachers (Chapters 7 and 9). 

1 1 . How well does the education of teachers prepare them to deal with the different 
facets of civic education? Teacher education or training programs often do 
not address civic education issues explicitly. The Phase 1 documents 
showed that, in some countries, teachers who have prepared to teach 
another subject have been asked to serve as teachers of civic education. 
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This Phase 2 volume provides data on the extent to which the teachers 
themselves believe that their training has prepared them adequately to 
teach topics relevant to civic education (reported in Chapter 9). 

12. How does the way in which schools are organized influence students' civic 

education ? The opportunities schools provide for meaningful participation, 
self-government and respect for rights are among the factors potentially 
influencing students’ attitudes and behaviors. Most countries’ Phase 1 
submissions highlighted aspirations to provide students with such 
experiences but few reported successful concrete initiatives. The idea that 
schools should be models of democracy is often stated but difficult to put 
into practice. Participation in the school as a community is covered in 
Phase 2 (Chapters 7 and 8). 



SUMMARY OF AIMS OF THE STUDY AND INFLUENCES 
ON IT 

This two-phased research study is intended to inform and stimulate discussion 
among policy-makers, curriculum developers, teachers, teacher educators, 
researchers and the general public. The study does not, however, try to identify 
a single best definition of citizenship or advocate a particular approach to civic 
education. Rather it tries to deepen the understanding of possibilities and 
practices in civic education as it takes place in different contexts. 

Although our conceptual model has focused the study’s attention on school- 
based, family, community and peer-group factors, the study is not an effort to 
refine theory. It has not been a curriculum development effort, although the 
test framework and the findings have implications for others who will develop 
curriculum, programs and materials in the future. 

Three major sources of influence have shaped this study. The first relates to the 
IEA organization and the member countries that chose to participate in it. 
Rigor and collaboration are the hallmarks of IEA studies. The rigorous 
standards for research developed by IEA over the past decades therefore have 
served as our standard. At several points we chose to narrow the focus of the 
study to ensure that we could meet the standards of rigor in instrumentation, 
sampling and analysis set by recent IEA studies, including TIMSS. The 
participating countries were collaborators in the design of the Civic Education 
Study, providing the International Coordinators and the International Steering 
Committee with advice about models, items and interpretations throughout 
the process. 

The second source of influence includes the theoretical frameworks and 
research literature not only in civic education but also in sociology, political 
science and developmental psychology. Some aspects of these frameworks are 
discussed in the sections of this chapter on the model (Figure 1.2), and others 
will become evident as the construction of particular scales is described in 
subsequent chapters. 

The policy questions guiding and linking both phases of the study are the 
third source of influence. Although formulated by the International Steering 
Committee five years ago, these questions remain important. We have collected 
data to address all of them. 
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It has not been possible in this volume to explore many questions interesting 
to policy-makers, educators and researchers. IEA will release the full set of data 
in 2002 for use by the research community, which will be able to conduct 
many additional analyses. To give only a few examples, those researchers who 
focus on a subset of countries may formulate scales using items that were left 
out of this volume because they could not be scaled to IEA standards across 
the full range of countries. Other researchers may form a broader measure of 
attitudes toward democratic values that includes opportunities for immigrants, 
women and ethnic minorities (dimensions that we separated or could not 
include). Still others may choose different methods for analyzing 
socioeconomic differences or school practices. 

The remainder of this volume provides analysis that is closely related to the 
original aims of the study, and suggests many directions that future analysis 
might take. 
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CHAPTER 2 




— 

HIGHLIGHTS RELATING TO METHODOLOGY 



• A review of documents submitted by 
countries during the first phase of the 
civic education study, together with 
extensive item writing, pre-pilot and pilot 
testing, and input from country 
representatives resulted in the 
development of an instrument requiring 
two class-hours to administer. This 
instrument meets IEA’s standards for 
psychometric quality. 



During 1999 nearly 90,000 students 
enrolled in the modal grade for 14-year- 
olds from 28 countries took the test of civic 
knowledge and skills, and the survey 
assessing concepts, attitudes and 
participatory actions. 



International assessment in civic education has been much less frequent than 
testing in other content areas in comparative education. More detailed 
information about instrument development is therefore contained in this report 
than would be required in frequently tested areas. 

In the first section of this chapter, we review the two-year process of 
identifying a common core of topics to form a content framework relating to 
citizenship and democracy valid across the 28 countries that participated in 
the civic education study. We also detail the three-year process of developing a 
fair and valid test (items designed with keys for correct answers) and survey 
(items assessing attitudes or beliefs for which there are no correct answers) to 
meet IEA standards. 

In the next section, we describe the study’s sampling. We chose the modal 
grade for 14-year-olds as the target population for two reasons. First, it is the 
standard IEA population, and it was the target population sampled in the 
1971 study of civic education (Torney, Oppenheim & Farnen, 1975). 

Secondly, and more importantly, some National Research Coordinators noted 
during the development of the 1999 plans that testing an older group meant 
facing substantial student drop-out. 

We devote the remainder of the chapter to a description of the international 
translation verification, testing, quality control and scaling. We present some 
characteristics of the achieved sample in a table, and summarize the modes of 
analysis and presentation. (For more detail, see the technical report of the 
study, Lehmann et al., forthcoming.) 

FRAMEWORK DEVELOPMENT DURING PHASE 1 

The Phase 1 national case studies were the basis for Phase 2 of the study, in 
particular providing the material from which the testing framework was 
developed. This framework is similar to the intended curriculum on which 
tests in other IEA studies have been based. 

The data collected during Phase 1 included summaries of what panels of experts 
in participating countries believed that 14-year-olds should know about 18 
topics relating to democratic institutions. These topics included elections, 
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individual rights, national identity, political participation, the role of the police 
and the military, organizations that characterize civil society, relation of 
economics to politics, and respect for ethnic and political diversity (Torney- 
Purta, Schwille & Amadeo, 1999). 

Early in the study it was clear that there was a common core of topics and 
concepts that experts in these countries believed 14-year-olds should 
understand. Following examination of Phase 1 material and a vote on these 
topics by the National Research Coordinators, the International Steering 
Committee chose three domains of clustered topics as ‘core international 
domains’. These were: 

Domain I. Democracy 

What does democracy mean, and what are its associated institutions and 
practices? The three sub-domains were: 

A) Democracy and its defining characteristics 

B) Institutions and practices in democracy 

C) Citizenship — rights and duties. 

Domain IT. National Identity, Regional and International Relationships 
How can the sense of national identity or national loyalty among young 
people be described, and how does it relate to their orientation to other 
countries and to regional and international organizations? The two sub- 
domains were: 

A) National identity 

B) International/ regional relations. 

Domain III: Social Cohesion and Diversity 

What do issues of social cohesion and diversity mean to young people, 
and how do they view discrimination? 

We also identified three other issues as important — the media, economics and 
local problems (including the environment) — but these were explored less 
systematically during Phase 2. 

As a next step in developing a content framework, personnel at the Phase 1 
Coordinating Center read the case study documents. They developed 
statements about what young people might be expected to know and believe 
about the three domains, and they elaborated on and illustrated these with 
quotations from the national case studies. This material formed the Content 
Guidelines for the International Test and Survey, which served as a concise 
statement of content elements in the three domains that were important across 
countries. The guidelines also provided a focus for those writing the test items. 
It was clear from the case study material that the greatest emphasis in the test 
should be on Domain I: Democracy, Democratic Institutions and Citizenship. 

In addition to giving input on content domains to be covered, the National 
Research Coordinators were involved in defining the types of items to include 
in the instrument: 

• Type 1 items: assessing knowledge of content. 

• Type 2 items: assessing skills in interpretation of material with civic or 
political content (including short text passages and cartoons). 

Types 1 and 2 items formed the test. These items had keyed correct answers. 
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Because civic education is an area where students’ content knowledge and 
skills are important but not the sole focus, the National Research Coordinators 
suggested three other item types: 

• Type 3 items: assessing how students understand concepts such as democracy 
and citizenship. 

• Type 4 items: assessing students’ attitudes (for example, feelings of trust in 
the government). 

• Type 5 items: assessing students’ current and expected participatory actions 
relating to politics. 

Types 3, 4 and 5 items formed the survey. These items did not have correct 
answers. 

Intersecting these five item types with the three study domains produced the 
following matrix, which served as the basis for the test and survey design. 



Item Type: 


1 


2 


3 


4 


5 


Domain I 

Democracy/ 

Citizenship 












Domain II 
National Identity/ 
International Relations 












Domain III 
Social Cohesion/ 
Diversity 













A little less than half of the testing time was devoted to a test including 
cognitive items that could be ‘keyed’ with correct and incorrect answers. A 
little less than half of the remaining testing time was devoted to a survey 
including non-keyed items that assessed concepts, attitudes and actions. The 
rest of the instrument asked about students’ perceptions of classroom climate 
and their confidence in participation at school, and obtained background 
information (including home literacy resources and the associations or 
organizations to which students belonged). A short period at the end of the 
second testing session was reserved for countries to administer nationally 
developed items. 

THE PROCESS OF TEST AND SURVEY DEVELOPMENT 
DURING PHASE 2 

Because there were no large existing sets of items that were likely to yield the 
number of items needed to fill in the matrix, extensive item writing was 
required. We began by reviewing materials in the Content Guidelines, other 
summaries of Phase I documents, and messages exchanged during an on-line 
conference on civic issues conducted with secondary school students in seven 
countries. We next invited all National Research Coordinators to submit items. 
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Our third task was to review the 1971 IEA Civic Education instrument, 
released items from United States and Canadian assessments, and the published 
research literature. Members of the International Steering Committee then 
wrote items, which were subsequently entered into an item database keyed to 
the content guidelines. Our fifth step involved asking groups of test specialists 
and content experts to review items in the database and their relation to the 
content framework. 

The result of this activity was the development of 140 knowledge and skills 
items (Types 1 and 2), each with one correct answer and four distracters, each 
of which was entered into the database for the 14-year-old population. All the 
items were suitable for administration in the participating countries. 

The items focused on principles, pivotal ideas and general examples, and not 
on the details of the political arrangements in any one country. For example, 
Type 1 /Domain I items covered the principles of democracy and its associated 
institutions across the countries participating in the study. The test did not 
include items about specific mechanisms of the electoral process or 
government structure in any particular country. The Type 1 / Domains II and 
III items likewise dealt with internationally relevant or generalized matters 
shared across countries. This emphasis differs from that in many national tests 
where items about each country’s political structure predominate. The IEA 
Civic Education Study Phase 2 items are congruent with information gathered 
during Phase 1 about what students are expected to know, and with recent 
expert statements such as that issued under the auspices of the Council of 
Europe about the role of history knowledge in civic education (Slater, 1995, 
146-48). 

Some of the Type 2 items (skills) asked students to distinguish between 
statements of fact and opinion. Others were based on a leaflet of the type 
issued during an election campaign, on the interpretation of a short article 
from a mock newspaper, or on a political cartoon. The general ideas for 
cartoons came from those published in newspapers. They were redrawn to 
communicate a single message that a 14-year-old across countries could be 
expected to understand. 

Pre-Piloting of Item Types 1 and 2 (Knowledge and Skills) 

Convenience samples of 14-year-olds in 20 countries were tested with 80 
items of Types 1 and 2. The National Research Coordinators discussed the 
content of the pre-pilot items and the test statistics at a meeting held in March 
1998. They agreed to retain 62 items, and prepared six items to fill gaps. 

Piloting of Item Types 1 and 2 (Knowledge and Skills) and the 
Resulting Final Test 

Between April and October 1998, 25 countries conducted pilot studies on 
Forms A and B of the test (Types 1 and 2 items described above) and survey 
(Types 3 through 5 items described below). In each country, judgement 
samples of about 200 students were tested (two class periods per student). The 
pilot countries included Australia, Belgium (French), Bulgaria, Chile, Chinese 
Taipei, Colombia, Cyprus, the Czech Republic, Estonia, Finland, Germany, 
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Greece, Hong Kong Special Administrative Region (SAR), Hungary, Italy, 
Latvia, Lithuania, Norway, Poland, Portugal, Romania, the Russian Federation, 
Slovenia, Switzerland and the United States. In addition to these countries, 
Denmark, England, the Slovak Republic and Sweden participated in the final 
testing of 14-year-olds. (Chinese Taipei was unable to obtain funding to 
continue past the pilot testing.) 

The National Research Coordinators were provided with item statistics for 
their countries, and they discussed each item within its content category at a 
November 1998 meeting. The small number of items that was unacceptable to 
one-fifth of the countries was dropped, in accordance with the rule used by 
IEA to promote test fairness across countries. Through a process of 
negotiation, the research coordinators chose, by consensus, 38 items of Types 
1 and 2 (knowledge and skills) from the 68 that had been piloted. The 
discrimination indices were greater than .30 for most items; coverage of the 
content framework and the research coordinators’ preferences were the 
decisive factors. 

The ratios of ‘number of items written’ to ‘number piloted’ to ‘number 
accepted’ were similar to IEA tests in other subject areas. Confirmatory factor 
analysis and IRT modeling, presented in Chapter 3, indicate a high-quality 
test across countries. These modern scaling methods (Frederiksen, Mislevy & 
Bejar, 1993) were our primary guide as we developed the test. Classical 
scaling methods also indicate a test of high quality. The alpha reliabilities for 
the final 38-item civic education test exceed .85 in each of the countries (see 
Chapter 3 and associated appendices for details). 

With respect to content coverage, within Domain I there are items covering all 
three sub-domains (definitions of democracy 6, democratic institutions 12, 
citizenship in democracy 12); within Domain II there are items covering the 
two sub-domains (national identification 2, international relations 3); within 
Domain III there are three items. Appendix Table A. 1 contains short 
descriptions of the 38 items and of the content categories in which they were 
classified, along with the percentage of students answering them correctly in 
the final test and the respective item parameters (discussed further in Chapter 3). 

Piloting of Item Types 3,4 and 5 (Concepts, Attitudes and Actions) 
and the Resulting Final Surveys 

The National Research Coordinators reviewed lists of suggested topics for 
Types 3 to 5 items and some prototype items at the March 1998 meeting. 
Most item sets for piloting were suggested by the research literature. Some 
revisions were necessary to adapt items originally designed for administration 
to adults in an interview, and ‘don’t know’ options were added. 

In mid- 1998 the research coordinators piloted the survey items along with 
two forms of the knowledge and skills test. Items for the survey were chosen 
through a process of negotiation similar to that described in the previous 
section. The final survey included 52 items of Type 3 (concepts), 62 items of 
Type 4 (attitudes) and 22 items of Type 5 (actions). Items assessing student 
background, school experience, organizational membership and peer group 
involvement were also included. Policy in some of the participating countries 
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prohibited questions about families’ social or political values, and no such 
items were included. The final test and survey were designed so that they 
could be administered in two class periods. The texts of all of the Types 3, 4 
and 5 items and of about half of the Types 1 and 2 items will be released for 
use by other researchers. 

Chapters 4 through 7 of this publication describe the rationale for items and 
scales included in the student survey, along with relevant research literature. 

The development of short survey instruments for teachers and for school heads 
(principals) began at the March 1998 meeting and covered the same content 
domains as the student instrument, along with questions about the school 
context and instruction. These instruments were piloted in the same countries 
and at the same time as the student instruments. The questions included and 
the results for the teacher survey are discussed in Chapter 9. The school 
questionnaire has been left for future international analysis and for national 
analysis. 

SAMPLING, TESTING AMD SCALING DUIRIMG PHASE 2 

Sampling from an Internationally Defined Population 

The internationally desired population was defined as follows: 

The population includes all students enrolled on a full time basis in that 
grade in which most students aged 14:00 to 14:1 1 [years; months] are 
found at the time of testing. Time of testing is the first week of the 8th 
month of the school year. 

In most cases testing took place between March and June 1999 in the 
Northern Hemisphere and between August and October 1999 in the Southern 
Hemisphere. In England and Sweden, testing was conducted in the second or 
third month of the school year because of the countries’ late entry into the 
study. In the United States the testing was done in the second month of the 
school year because of uncertainty as to the age distribution of students in the 
eighth month of the year (resulting from the varying school entry dates set by 
districts). 

In the majority of countries, Grade 8 was selected. In nine countries, Grade 9 
was chosen. In Switzerland, differences between regions led to the selection of 
Grades 8 or 9, depending on the structure of the educational system. In 
Portugal, Grade 8 was selected even though the proportion of 14-year-olds in 
this country tends to be slightly higher in the adjacent Grade 9. The average 
age of respondents in the selected Grade 8 was 14:5, which was similar to the 
average age in most other countries in this study. If Grade 9 had been used in 
Portugal, the average age would have been 15:4. 

In two countries (Hong Kong/SAR and the Russian Federation), the average 
age was above 1 5:00 and therefore did not meet the study’s age/ grade 
specifications. In two countries (Belgium/French and Chile), the average age 
was between 14:00 and 14:11, but the proportion of 13-year-old students in 
the tested grade ended up being slightly higher than the proportion of 14- 
year-old students. 
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In Germany, three federal states (‘Bundeslander’) refused to participate in this 
study, and one federal state did not permit testing in high schools 
(‘Gymnasium’). Therefore, the sample was not representative for the population 
of all 14-year-old students in this country but only for those in the 
participating federal states. 

A two-stage stratified cluster design for sampling was employed in 
consultation with IEA sampling experts. At the first stage, schools were 
sampled using a probability proportional to size (PPS). 1 At the second stage 
the sample consisted of one intact classroom per school from the target grade. 
The chosen class was not to be tracked by ability and was, where possible, to 
be in a civic-related subject (for example, history, social studies). 

Table 2.1 shows the participation rates of the 28 countries. National Research 
Centers made every attempt to meet the sampling requirements, but in some 
countries there was resistance from teachers and schools. Ten countries failed 
to reach a 7 5 percent overall participation rate before replacement as specified 



Table 2.1 Participation Rates and Sample Sizes 



Country 


School 

Participation 

Before 

Replacement 

(Weighted 

Percentage) 


School 

Participation 

After 

Replacement 

(Weighted 

Percentage) 


Total 

Number of 
Schools That 
Participated 


Student 

Participation 

Rate 


Total 

Number of 
Students 
Assessed 


Overall 
Participation 
Rate Before 
Replacement 


Overall 
Participation 
Rate After 
Replacement 


Australia 


75 


94 


142 


92 


3331 


69 


86 


Belgium (French) 


57 


75 


112 


93 


2076 


53 


70 


Bulgaria 


86 


86 


148 


93 


2884 


80 


80 


Chile 


98 


100 


180 


97 


5688 


94 


97 


Colombia 


66 


94 


144 


96 


4926 


64 


90 


Cyprus* 


100 


100 


61 


96 


3106 


96 


96 


Czech Republic 


91 


99 


148 


95 


3607 


86 


94 


Denmark 


71 


71 


178 


93 


3208 


66 


66 


England 


54 


85 


128 


93 


3043 


50 


79 


Estonia 


84 


85 


145 


90 


3434 


76 


77 


Finland 


93 


98 


146 


93 


2782 


86 


91 


Germany 


63 


94 


169 


89 


3700 


56 


84 


Greece 


88 


93 


142 


97 


3460 


. 85 


90 


Hong Kong (SAR) 


90 


100 


150 


99 


4997 


89 


99 


Hungary 


99 


99 


146 


95 


3167 


94 


94 


Italy 


93 


100 


172 


96 


3808 


89 


96 


Latvia 


89 


91 


130 


91 


2572 


81 


82 


Lithuania 


93 


97 


169 


90 


3494 


84 


87 


Norway 


75 


77 


154 


93 


3321 


70 


71 


Poland 


83 


90 


179 


94 


3376 


78 


84 


Portugal 


98 


99 


149 


95 


3261 


93 


95 


Romania 


97 


97 


146 


99 


2993 


96 


96 


Russian Federation 


96 


98 


185 


97 


2129 


94 


95 


Slovak Republic 


79 


97 


145 


94 


3463 


74 


91 


Slovenia 


93 


99 


149 


96 


3068 


89 


95 


Sweden 


93 


94 


138 


94 


3073 


88 


88 


Switzerland 


71 


87 


157 


97 


3104 


69 


84 


United States 


65 


83 


124 


93 


2811 


61 


77 


* In Cyprus two classes per school were sampled. 



Source: IEA Civic Education Study ; Standard Population of 14-year-olds tested in 1999. 
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in the sampling guidelines. In Belgium (French), Denmark and Norway, the 
overall participation rate, even after replacement of schools, was lower than 75 
percent. 2 Student participation rates were at least 89 percent in all participating 
countries, however. 

Sample sizes of schools per country varied between 112 and 185. In Cyprus, 
all 61 schools in the country were tested, and from each school two classes 
were sampled. Student sample sizes ranged between 2,076 and 5,688. In some 
countries, disproportional samples were drawn (for example, to include larger 
sub-samples of specific school types). Sampling weights were applied. 

Table 2.2 summarizes three basic characteristics of the sample by country: 
mean and standard deviation of the age of students tested, the percentage of 
females, and the percentage of students who answered that they had not been 
born in the country. The age distribution is discussed in Chapter 3. The most 
serious gender disparity was in Colombia, where 58 percent of the student 



Table 2.2 Sample Characteristics 



Country 


Age 


Percentage of 
Females 


Percentage of 
Students Not 
Born in Country 


Mean 


Standard 

Deviation 


Percentage of 
14-year-olds 


Australia 


14.6 


0.5 


67 


55 


10 


Belgium (French) 


14.1 


0.7 


34 


49 


10 


Bulgaria 


14.9 


0.6 


59 


52 


4 


Chile 


14.3 


0.8 


40 


49 


2 


Colombia 


14.6 


1.2 


35 


58 


3 


Cyprus 


14.8 


0.4 


75 


51 


9 


Czech Republic 


14.4 


0.4 


70 


51 


2 


Denmark 


14.8 


0.4 


66 


49 


7 


England 


14.7 


0.3 


79 


50 


6 


Estonia 


14.7 


0.6 


67 


52 


6 


Finland 


14.8 


0.3 


67 


52 


3 


Germany 1 


n.a 


n.a 


n.a 


51 


19 


Greece 


14.7 


0.5 


83 


52 


6 


Hong Kong (SAR) 


15.3 


0.8 


38 


49 


20 


Hungary 


14.4 


0.5 


70 


50 


3 


Italy 


15.0 


0.7 


58 


52 


2 


Latvia 


14.5 


0.6 


62 


52 


5 


Lithuania 


14.8 


0.6 


67 


51 


3 


Norway 


14.8 


0.3 


71 


51 


6 


Poland 


15.0 


0.4 


54 


52 


1 


Portugal 


14.5 


1.0 


35 


52 


5 


Romania 


14.8 


0.5 


65 


48 


1 


Russian Federation 


15.1 


0.5 


48 


53 


14 


Slovak Republic 


14.3 


0.4 


69 


53 


2 


Slovenia 


14.8 


0.4 


74 


50 


4 


Sweden 


14.3 


0.4 


79 


52 


8 


Switzerland 


15.0 


0.7 


55 


51 


17 


United States 


14.7 


0.6 


74 


51 


11 


International Sample 


14.7 


0.7 


62 


51 


7 



1 Information on age is not available for Germany. International sample figures based on 27 countries. 



Source: IEA Civic Education Study, Standard Population of 14-year-olds tested in 1999. 




CHAPTER 2 METHODOLOGY 




35 



respondents were female. The proportion of students not born in the country 
ranged from 1 percent (in Poland and Romania) to 19 and 20 percent in 
Germany and Hong Kong (SAR), respectively. This matter is discussed further 
in Chapter 5. 

Instrument Translation 

The pilot and final instruments were prepared in English and distributed by 
the International Coordinating Center (ICC). The National Research Centers 
then translated them into 22 languages. The ICC developed guidelines and 
detailed translation notes indicating alternative wordings adapted to the 
country’s specific context. 

Translated instruments for the final testing had to be submitted to the ICC for 
verification. Native speakers with a very good command of English and no 
working relationship with the National Research Centers verified the 
translations of school, teacher and student instruments according to guidelines 
issued by the ICC. The results of this verification were returned to the 
National Research Coordinators. In most countries, suggestions for 
improvement were taken into account in the final translations. In only three 
countries were translations not submitted in time for this process to take place. 
However, in these cases, the verifications did give some after-the-fact control 
over deviations (which were, in fact, few in number). 

This process together with the translation verification of the pilot instruments 
in 1998 provided a high degree of quality control in this area. For 24 of the 
28 countries the instruments were verified twice. For the four countries that 
entered the study after the pilot, there was only one verification. Instruments 
from English-speaking countries did not require translations but were reviewed 
for modifications necessary to adapt them to each country’s political and 
cultural context. 

Data Collection and Quality Control for Testing 

Each participating country was responsible for data collection. Manuals for 
field operations, the school coordinators and the test administrators, together 
with tracking forms were adapted by the IEA Data Processing Center from 
those developed for TIMSS. The distribution of this material to the National 
Research Centers was carried out with the cooperation of the ICC. Where 
necessary, the manuals were translated into the country’s language. Data 
collection at the schools followed strict guidelines for test administration and 
timing to safeguard comparability across countries. Full confidentiality of 
responses was guaranteed. Data entry was conducted by the National Research 
Centers. 

The National Research Coordinators were asked to make follow-up calls on 
the day after testing to a 25 percent random sample of the tested schools. 

They were instructed to ask about deviations from testing procedures (using 
guidelines provided by the ICC). In a few countries, organizational problems 
made this task impossible, but every effort was made to examine these data 
with special care. In some countries, the national centers set up additional 
control monitoring. 

38 
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After completion of the testing, the National Research Centers responded to a 
questionnaire on quality control that could be used as an additional check. 
Most centers completed this questionnaire within one month of testing. 

Data Processing and Weighting 

After collection, the data sets were submitted in a standard format to the IEA 
Data Processing Center (DPC) in Hamburg, which created the international 
database for the study. The DPC compared the database to the school, 
classroom, teacher and student tracking forms completed during data 
collection. They also checked and double-checked the data for inconsistencies. 
All deviations were documented and sent to the National Research Centers for 
clarification. The data-cleaning process consisted of several steps designed to 
guarantee high quality. The DPC also computed the weights to be applied to 
the sample according to the previously approved sampling design in each 
country (in line with IEA guidelines). 

Confirmatory Factor Analysis and IRT Scaling 

Structural equation modeling (SEM), including confirmatory factor analysis 
(CFA), was used to confirm theoretically expected dimensions or to re-specify 
the dimensional structure of the instruments. These procedures take into 
account the measurement error associated with indicators, providing more 
reliable estimates for latent variables and scales than classical psychometric 
methods. 3 

For both multiple-choice and categorical items, item response theory (IRT) 
scaling methods were used. For the cognitive test, a one-parameter Rasch 
model was fitted to the data; for attitudinal items the partial credit model was 
applied. The mean Rasch score for the scores derived from the test was set at 
100, with a standard deviation of 20. The mean Rasch score for the scales 
derived from the survey (measuring concepts, attitudes and actions) was set at 
10, with a standard deviation of 2. 

There were several reasons for using IRT scaling. In this study all students 
were administered exactly the same 38-item test, so IRT scaling was not 
absolutely necessary (as it would have been if there had been a larger set of 
items from which several test forms had been constructed). In the case of the 
test items, however, the Rasch method provided a common scale for all 
countries, allowing the exclusion of items that did not fit the model in a few 
countries and without jeopardizing the comparability of the international 
scale. This method is prescribed in IEA studies. 

Attitudinal items that had missing values, resulting from students who 
answered ‘don’t know’ or who left items out, was a potential problem. Here, 
IRT scaling provided an elegant way of computing estimates for latent 
dimensions, even those with missing information. 
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The analysis of the international data consisted of several steps: 

• Computation of item statistics for all test and survey items. 

• Exploratory factor analysis and computation of classical scale reliabilities for 
the theoretically expected scales for each country. 

• Confirmatory factor analysis with structural equation modeling on an 
international random sample of 200 students from each country, followed 
by the checking of models for each country. 

• Selection of scales based on theoretical and empirical grounds. 

• Estimation of Rasch models for the selected scales on an international 
random sample of 200 students per country. 

• Item adjudication to examine scales by country and to make further 
refinements. 

The final scaling used a calibration sample of 500 students selected randomly 
from weighted country data. Item parameters were estimated for the 
calibration sample and used as anchors for subsequent scaling of country data 
sets. 

Complex sampling (such as the multi-stage sampling used here) makes simple 
random sampling formulas for estimating standard errors inadequate. In order 
to estimate correct sampling errors for each statistic in this report, we applied 
the ‘jack-knife’ procedure. The overall estimate of a sample statistic plus or 
minus two standard errors gives a 95 percent probability of inferring the 
correct mean in the population based on the student sample. 



Many of the scales and items in the test and the survey were derived from 
previous research and, in some cases, had been the subject of extensive debate 
and empirical study by political or educational researchers, sociologists and 
psychologists. An attempt has been made in the panels in Chapters 3-7 to 
briefly review the methods and a selection of the findings of previous cross- 
national research, especially that conducted in countries participating in this 
study. 

The 38 multiple choice items in the test of knowledge and skills have correct 
answers. The IRT scaling process for these items is covered in Chapter 3. The 
Rasch scores presented in this chapter were normed to have an international 
mean of 100 and a standard deviation of 20. The scores based on these items 
are presented in the same format as in many other IEA studies, with the 
countries in rank order by score. 

The items in the survey of concepts, attitudes and actions do not have correct answers. 
These results are presented in Chapters 4-7. The large majority of items were 
statements to which the student was to respond on a four-point scale with an 
additional ‘don’t know’ option. The labeling of scale points differed. For many 
scales they were ‘strongly agree’, ‘agree’, ‘disagree’, ‘strongly disagree’. Other 
response formats asked about how important something was or how 
frequently something happened. 
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The examination of confirmatory factor analyses by the International Steering 
Committee led to the choice of 1 1 sets of survey items for scaling. An IRT 
scaling procedure was applied at the International Coordinating Center to 
each set of items, and the resulting Rasch scales were normed to have an 
international mean of 10 and a standard deviation of 2. These scores allow 
statistically sound comparisons between countries’ means and the international 
mean, as well as between one country’s mean and that of another. Appendix B 
includes an item-by-score map for each concept, attitude or action scale in 
Chapters 4—7, allowing the reader to ascertain the response on the four-point 
scale that corresponds to Rasch scores from 4 to 16. Although country means 
were in the range from 8 to 12, there were student respondents in each 
country with scores well below 8, and others that were well above 12. The 
Appendix B material also includes the percentage distribution of responses to 
each scaled item based on the entire group of countries. 

The figures in Chapters 4—7 present country means on these scaled scores. All 
countries appear in alphabetical order, with a confidence bound for each mean 
of two standard errors. An upward or downward arrow also appears to indicate 
whether a country’s mean is significantly higher or lower than the 
international mean. Chapter 10 summarizes these differences across all scales 
and countries. 

A figure comparing country means is included for each of the Rasch scores in 
Chapters 4—7. An additional figure is included to illustrate gender differences 
only for those scales where half or more countries show a significant gender 
difference (p < .05 with a Dunn-Bonferroni correction for multiple 
comparisons). If fewer than half of the countries show a significant difference, 
the text lists the gender differences that are statistically significant at the .05 
level, but no separate figure is included. 

SUMMARY OF METHODS 

The following were used to develop the two class-hours of the 1999 IE A 
Civic Education Study test and survey: 

• an iterative process of review of Phase 1 documents submitted by countries; 

• references to the research and theoretical literature; 

• extensive item writing; 

• review by experts internationally and within participating countries; 

• pre-pilot and pilot testing; 

• item choice by participating countries. 

The test and survey was administered to nationally representative samples 
totaling nearly 90,000 14-year-old students in 28 countries. Confirmatory 
factor analysis and Rasch scaling were used to develop scales. Much of the 
data is presented in this volume in figures that allow an analysis of countries’ 
positions significantly above, not significantly different from, or significantly 
below the international mean. 

A similar process was undertaken for the development of the Teacher 
Questionnaire and a very short School Questionnaire (covered in Chapter 9). 
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Quality control procedures were undertaken, including the review of samples 
by sampling experts, two translation verifications by independent experts, and 
other measures prescribed by the IEA technical standards. Panel 2.1 presents a 
listing of these quality control procedures. 



PANEL 2.1 Quality Control Processes 



Control Processes Anchored in the Study's Conceptualization 

The study’s design, instruments and reports have been: 

• Connected to 15 policy questions formulated to guide Phases 1 and 2. 

• Referenced to 18 case study framing questions in Phase 1. 

• Framed by the ‘octagon model’ and ‘situated cognition perspective’ in Phases 1 and 2. 

• Anchored in Phase 1 country case study data (leading to definition of the three domains 
for Phase 2). 

• Scaffolded by content guidelines (including quotations from Phase 1 documents). 

• Organized under three domains forming the test and survey framework. 

• Referenced to the research literature on political attitudes in youth and adults and to 
theories of democracy. 

• Built using five item types matched to country expectations identified during Phase 1. 

Control Processes Relating to IEA Standards and Participating Country Input 

The study’s design, instruments and reports have been: 

• Guided and judged by IEA technical standards and procedures (for example, regarding 
sampling and testing). 

• Influenced by participating countries’ input (through National Expert Panels and 
National Research Coordinators forming a de-centered network for test adaptation). 

• Shaped by the analysis of pre-pilot psychometric data for test items from 20 countries 
interpreted by National Research Coordinators. 

• Shaped by the analysis of pilot psychometric data for test and survey from 25 countries 
and interpreted by National Research Coordinators. 

• Shaped in meetings between National Research Coordinators and Data Processing Center 
Personnel (regarding sampling, weighting of samples and data submission). 

• Reviewed periodically by the IEA Technical Executive Group. 

• Informed by independent verification of test translations and of concept equivalence (of 
the pilot and the final test and survey). 

• Monitored using National Research Coordinators’ reports regarding the testing process. 

• Referenced to an analysis plan guided by policy questions and IEA principles. 

• Finalized through the International Steering Committee’s review of Rasch scaling for test 
and survey items and choice of scales to be reported. 
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NOTES 



1 The general procedure followed closely the one adopted for the Third International 
Mathematics and Science Study (TIMSS) as described in Foy, Rust and Schleicher (1996). 
In some countries, the sample for the Civic Education Study was linked to the TIMSS-R 
(Repeat), which was done in the same year as the Civic Education Study. 

2 Originally sampled schools that refused to participate could be replaced by additionally 
sampled schools. In Denmark no replacement schools were sampled. 

3 We have included some classical psychometric indices in appendices because some readers 
may be more familiar with them. 
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CHAPTER 3 




/ 

HIGHLIGHTS RELATING TO CIVIC KNOWLEDGE 



• The results show that it is possible to 
construct a meaningful, reliable and valid 
international test of student knowledge 
about democratic institutions, principles, 
processes and related topics despite 
differences in the political systems in 
different participating countries. 

• The differences between countries in 
mean performance on this test are in 
general not large. Twenty-five of the 28 
countries differ by less than half a 
standard deviation from the international 
average. 

• Students’ abilities to answer questions 
requiring knowledge of civic content 
and questions requiring skill in 
interpreting civic-related material are 
distinguishable empirically. Although 
this distinction between content 
knowledge and skills in interpretation 
has a very limited influence on the 
countries’ rankings, it adds to the 
understanding of countries’ specific 
strengths and weaknesses. 

V 



Unlike studies of earlier decades, this 
study reveals no significant differences in 
mean performance between boys and 
girls in 27 of the 28 countries, when the 
comparison is made without controlling 
for other variables. 

Students reporting more home literacy 
resources consistently do better on the 
test. 

Over 75 percent of the students in most 
participating countries are able to answer 
questions dealing with the fundamental 
nature of laws and political rights. 
However, few students in the 
participating countries can answer more 
demanding questions on the test that 
have to do, for example, with deciding 
between election candidates based on 
their policy positions, understanding 
processes of political reform, and 
grasping the implications of economic 
and political choices made by policy- 
makers. 



The IEA Civic Education Study faced considerable skepticism about the 
possibility of developing a valid test to measure civic knowledge across diverse 
political systems. Chapter 2 has described the process by which we met this 
challenge, namely producing a test rooted in the content domains defined 
collaboratively in Phase 1 of the study, meeting IEA technical standards and 
allowing for extensive procedures of quality control. For a review of the 
previous research in this area, see Panel 3.1 

In this chapter, we present a summary of the results achieved on this test by 
nearly 90,000 students who constituted nationally representative samples from 
the 28 participating countries. We describe the methods applied in the scoring 
and scaling of student responses, and then display the distribution of test 
scores by country, followed by an analysis of how the test can be partitioned 
into two dimensions relating to knowledge of content and skills in 
interpreting civic-related information. Finally, we deal with the ways in which 
the key variables of gender and home literacy resources are related to civic 
knowledge. In short, the chapter provides a base for the comparative 
assessment of what students in participating countries know about the nature 
and workings of democracy and, to some extent, about the other two core 
content domains of the study. 
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PANEL 3.1 Previous Research on Students’ Civic Knowledge 



First, it is important to acknowledge the role of general literacy in acquiring political knowledge. For 
example, Chall and Henry (1991) noted that considerably more than a minimal level of literacy is 
required for understanding documents such as constitutions or for locating information in sources such 
as newspapers. 

As part of a more specific look at the role of knowledge in the context of civic education, the 197 1 
IEA Civic Education Study used a test of 47 items for 14-year-olds (Torney, Oppenheim & Farnen, 
1975). In this study, students in the Federal Republic of Germany had the second highest cognitive 
score. The United States ranked fourth, Finland ranked fifth, and Italy ranked seventh (out of eight 
countries). Among other countries that participated in the 1971 but not in the 1999 study, students 
from the Netherlands ranked first, those from Israel ranked third, those from New Zealand ranked 
sixth, and those from Ireland ranked eighth. At age 14, males performed higher on the test than 
females in the Federal Republic of Germany, Finland and the United States (but not in Italy) in a 
comparison similar to that reported in this chapter. Those from higher socioeconomic backgrounds 
performed at a higher level in all the countries, although the differences were especially large in the 
United States (Torney etai, 1975, pp.138, 156). The encouragement of independent expression of 
opinion in the classroom was a positive predictor in all the countries (p.140). 

In some countries there have also been large-scale national assessments of civic knowledge. In the 
United States a National Assessment of Educational Progress (NAEP) regularly tests students at Grades 
4, 8 and 12 (ages approximately 9, 13 and 17) in civic-related content areas. Multiple choice items are 
used, along with items that require students to write a response (sometimes relating to a picture of a 
historical event, cartoon or newspaper article). No attempt is made to separate performance on 
knowledge of content and skills in interpretation (as in the current IEA study). Proficiency levels for 
the ‘total knowledge’ scores are set by experts. Most students are classified as having ‘basic’ rather than 
‘proficient’ or ‘advanced’ understanding (Lutkus, Weiss, Campbell, Mazzeo & Lazer, 1999; Torney- 
Purta, 2000). There is a substantial gap between the scores of students from more and less affluent and 
highly and less educated home backgrounds (Niemi 8c Junn, 1998). 

Gender differences in the NAEP have been variable. In the 1988 assessment, males tended to perform 
at a somewhat higher level than females. Gender differences were especially pronounced in knowledge 
of political parties, elections and protest activities. In the 1998 assessment, these differences were either 
very small or showed females to have slight superiority. The most comprehensive recent analysis of 
school-based predictors of achievement in NAEP, by Niemi and Junn (1998, using the 1988 data), 
found that frequent testing seemed to be counterproductive in students’ learning of civic content. The 
taking of classes in which civic topics were studied and where participation in role-playing elections or 
mock trials was included seemed to have a positive effect. 

A study in Australia with 1,000 students from Years 5 and 9 tapped political understanding by asking 
questions to which students wrote answers (Doig, Piper, Mellor & Masters, 1993/94). Topics included 
the meaning and origin of laws, the electoral process, influences on political decisions, processes of 
enactment and implementation of parliamentary decisions, and the meaning of democracy. Each 
response was scored as being at one of several levels ranging from simplistic, vague or confused to 
sophisticated and complex (including the ability to apply principles). The average student was found to 
be able to ‘recognize key aspects of democracy in a generalized way’. On average, Year 9 students had 
higher scores than Year 5 students. Females at Year 5 had higher scores than males, but there were no 
significant gender differences among the older students. 

For a fuller review of studies in these areas, see Torney-Purta, Hahn and Amadeo (2001). 
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CIVIC KNOWLEDGE IN THE 1999 IEA INSTRUMENT: 

HOW IT WAS MEASURED 

The IEA civic knowledge test consists of 38 items, 25 of which refer to 
knowledge of content (Type 1) and 13 to skills in interpretation (Type 2). All 
items were given in a multiple-choice format, with student responses coded as 
correct or incorrect. The items cover a broad range of content areas selected 
from a much larger set of trial items after intensive piloting (see Chapter 2). In 
the final test version, the international average of correct answers was 64 
percent, which indicates that, for the majority of students, the test was far from 
being too difficult. The results of Rasch scaling as well as classical item 
statistics (Appendix Tables C. 1 and C.2) show that the test has satisfactory 
characteristics; for example, alpha coefficients are at least .85 in all countries. 
More detail will appear in the technical report (Lehmann etal., forthcoming). 

SAMPLE ITEMS AND THEIR RESULTS 

To convey a notion of what was measured by the test, five examples are given 
and briefly discussed here (refer to Figures 3.1a-e). 



Figure 3.1a Item Example: Identify a non-democratic government 



Country 


Correct Answers 

(in %) 


Example 3 (Item #17) 
Type 1: Knowledge of Content 


Australia 


50 ( 1 . 3 ) 




Belgium (French) 


51 ( 1 . 8 ) 




Bulgaria 


53 ( 2 . 0 ) 




Chile 


44 ( 1 . 3 ) 




Colombia 


38 ( 1 . 9 ) 




Cyprus 


59 ( 1 . 3 ) 


17. Which of the following is most 


Czech Republic 


60 ( 1 . 6 ) 


likely to cause a government to 


Denmark 


46 ( 1 . 1 ) 


be called non-democratic? 


England 


45 ( 1 . 1 ) 




Estonia 


39 ( 1 . 4 ) 


A. People are prevented from 


Finland 


63 ( 1 . 3 ) 


criticising the government.* 


Germany 


56 ( 1 . 2 ) 


B. The political parties criticise 


Greece 


67 ( 1 . 2 ) 


each other often. 


Hong Kong (SAR) 


73 ( 1 . 3 ) 


C. People must pay very high taxes. 


Hungary 


45 ( 1 . 2 ) 


D. Every citizen has the right to a 


Italy 


63 ( 1 . 5 ) 


job. 


Latvia 


36 ( 1 . 9 ) 




Lithuania 


44 ( 1 . 6 ) 




Norway 


57 ( 1 . 0 ) 




Poland 


65 ( 2 . 3 ) 




Portugal 


55 ( 1 . 5 ) 




Romania 


42 ( 1 . 8 ) 




Russian Federation 


57 ( 2 . 3 ) 




Slovak Republic 


60 ( 1 . 6 ) 




Slovenia 


50 ( 1 . 3 ) 




Sweden 


66 ( 1 . 6 ) 




Switzerland 


56 ( 1 . 6 ) 




United States 


53 ( 1 . 7 ) 




International Sample 


53 ( 0 . 3 ) 





( ) Standard errors appear in parentheses. 
* Correct answer. 



Source: IEA Civic Education Study, Standard Population of 14-year-olds tested in 1999. 
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Figure 3.1b Item Example:This is the way history textbooks are sometimes written 



Country 


Correct Answers 




(in %) 


Australia 


75 (1.2) 


Belgium (French) 


66 (2.1) 


Bulgaria 


47 (2.3) 


Chile 


49 (1.5) 


Colombia 


48 (2.3) 


Cyprus 


53 (1.1) 


Czech Republic 


54 (1.5) 


Denmark 


60 (1.0) 


England 


76 (1.2) 


Estonia 


39 (1.2) 


Finland 


65 (1.3) 


Germany 


61 (0.9) 


Greece 


56 (1.3) 


Hong Kong (SAR) 


76 (1.4) 


Hungary 


67 (1.3) 


Italy 


61 (1.3) 


Latvia 


48 (1.7) 


Lithuania 


48 (1.4) 


Norway 


49 (1.0) 


Poland 


64 (2.1) 


Portugal 


49 (1.1) 


Romania 


26 (1.7) 


Russian Federation 


45 (2.1) 


Slovak Republic 


72 (1.5) 


Slovenia 


56 (1.1) 


Sweden 


52 (1.2) 


Switzerland 


67 (1.4) 


United States 


79 (1.4) 


International Sample 


57 (0.3) 



Example 5 (Item #36) 
Type 2: Skills in Interpretation 




36. What is the message or main point of this cartoon? 
History textbooks ... 

A. are sometimes changed to avoid mentioning 
problematic events from the past.* 

B. for children must be shorter than books written for 
adults. 

C. are full of information that is not interesting. 

D. should be written using a computer and not a pencil. 



( ) Standard errors appear in parentheses. 
* Correct answer. 



Source: IEA Civic Education Study, Standard Population of 14-year-olds tested in 1999. 



The first example (Figure 3.1a; see also example 3 in Figure 3.2) is a typical 
Type 1 item. It requires respondents to demonstrate knowledge of content by 
identifying a practice that ‘most likely causes a government to be called non- 
democratic’. Figure 3.1a displays the question and the four answers from 
which the students had to choose. Among these four answers, the first one (A: 
‘People are prevented from criticising the government’) is the correct response. 
The percentages of correct answers range from 36 to 73 percent. The average 
of correct answers across all countries (equally weighted) is 53 percent. If this 
international mean is compared with the overall percentage of correct answers 
in the test (64 percent), it is clear that the item is relatively difficult, although 
certainly within reach for most of the students. The correct answer requires a 
reliable knowledge base as to the properties of democratic governments and 
the ability to apply that knowledge to the opposite case (‘non-democratic 
government’). 
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The second example (Figure 3.1b; example 5 in Figure 3.2) is a Type 2 item that 
is intended to measure skills in the interpretation of civic-related material. Figure 
3.1b demonstrates how the item was presented. There is a cartoon showing 
someone erasing words from a book, presumably one on the history of a nation as 
suggested by a flag and the word ‘history'. The students were asked about the 
message or main point of this cartoon and had to select the correct response 
(A: ‘History textbooks are sometimes changed to avoid mentioning problematic 
events from the past'). The distribution of correct answers across countries ranges 
from 26 to 79 percent. The international mean of 57 percent correct answers 
shows that this item is less difficult for students than the previous example, 
although it is slightly harder than the test on average. The task relates to the 
domain of national identity and international relations and requires the ability to 
interpret or comprehend the message that the cartoonist has attempted to convey. 



Figure 3.1c Item Example:This is an election leaflet 



Country 


Correct Answers 

(in %) 


Example 6 (Item #23) 
Type 2: Skills in Interpretation 


Australia 


78 (1.3) 




Belgium (French) 


56 (1.8) 




Bulgaria 


47 (2.4) 








Chile 


54 (1.5) 




We citizens have had enough! 




Colombia 


40 (2.4) 




A vote for the Silver Party means a vote for 




Cyprus 


81 (0.9) 




higher taxes. 




Czech Republic 


66 (1.6) 




It means an end to economic growth and a waste 




Denmark 


49 (1.1) 




a f /Aiir n oft a tA focAiirroc 




England 


75 (1.2) 




or our nation s rcsourccSi 




Estonia 


54 (1.4) 




Vote instead for economic growth and 




Finland 


85 (0.8) 




free enterprise. 




Germany 


81 (0.9) 




Vote for more money left in everyone’s wallet! 




Greece 


73 (1.3) 




Let’s not waste another 4 years! 




Hong Kong (SAR) 


76 (1.4) 




VOTE FOR THE GOLD PARTY. 




Hungary 


78 (1.2) 








Italy 


85 (1.2) 




Latvia 


44 (1.9) 


23. This is an election leaflet which has probably been 


Lithuania 


55 (1.6) 


issued by ... 


Norway 


57 (0.9) 








A. the Silver Party. 


Poland 


58 (2.0) 




Portugal 


55 (1.3) 


B. a party or group in opposition to the Silver Party.* 






C. a group which tries to be sure elections are fair. 


Komania 

Russian Federation 


4b tZ.UJ 
45 (1.9) 


D. the Silver Party and the Gold Party together. 


Slovak Republic 


66 (1.6) 




Slovenia 


75 (1.0) 




Sweden 


73 (1.5) 




Switzerland 


77 (1.3) 




United States 


83 (1.4) 




International Sample 


65 (0.3) 





( ) Standard errors appear in parentheses. 
* Correct answer. 



Source: IEA Civic Education Study, Standard Population of 14-year-olds tested in 1999. 
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Figure 3. Id Item Example: Importance of many organisations for democracy 



Country 


Correct Answers 


Example 7 (Item #07) 




(in %) 


Type 1: Knowledge of Content 


Australia 


78 (1.2) 




Belgium (French) 


68 (1.6) 




Bulgaria 


71 (1.9) 




Chile 


69 (1.1) 




Colombia 


60 (2.0) 




Cyprus 


80 (1.1) 


7. In a democratic country [society] 


Czech Republic 


76 (1.2) 


having many organisations for 


Denmark 


75 (0.9) 


people to join is important 


England 


79 (1.0) 


because this provides ... 


Estonia 


61 (1.1) 


A. a group to defend members who 


Finland 


82 (1.0) 


Germany 


67 (1.0) 


are arrested. 


Greece 


76 (0.9) 


B. many sources of taxes for the 


Hong Kong (SAR) 


79 (1.1) 


government. 


Hungary 


46 (1.3) 


C. opportunities to express different 








Italy 


71 (1.4) 


points ot view. 


Latvia 


55 (1.8) 


D. a way for the government to tell 


Lithuania 


61 (1.4) 


people about new laws. 


Norway 


69 (0.9) 




Poland 


78 (1.6) 




Portugal 


59 (1.2) 




Romania 


48 (2.2) 




Russian Federation 


68 (1.6) 




Slovak Republic 


75 (1.1) 




Slovenia 


62 (1.1) 




Sweden 


70 (1.5) 




Switzerland 


68 (1.3) 




United States 


78 (1.4) 




International Sample 


69 (0.3) 





( ) Standard errors appear in parentheses. 
* Correct answer. 



Source: IEA Civic Education Study, Standard Population of 14-year-olds tested in 1999. 



The third example (Figure 3. 1 c; example 6 in Figure 3.2) is also a Type 2 item, in 
this case referring to institutions and practices in democracy. Here, students were 
asked to interpret an electoral leaflet directed against an imaginary party 
(presumably in power) and to indicate which political group had probably issued 
it. The correct answer is B (‘a party or group in opposition to the Silver Party’). 
The lowest proportion of correct answers found in any country is 40 percent and 
the highest 85 percent. The international average is 65 percent. Thus, the item is 
slightly easier than the international average of the test as a whole. There are 
several clues suggesting the correct solution, although the interpretation of some 
of these requires quite complex inferences as to the two mentioned parties’ 
approaches to taxation and government spending. The last line of the leaflet (‘Vote 
for the Gold Party’) is unambiguous and clearly marks its origin. As such, the 
interpretative task is primarily to identify the alleged negative economic 
consequences of the Silver Party’s fiscal policies and to recognize that such 
arguments could come only from an opposing group. 
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The fourth example (Figure 3. Id; example 7 in Figure 3.2) is a Type 1 item, 
aiming at civic-related content knowledge, more specifically knowledge as to the 
rights and duties of citizens in a democratic country. The students could choose 
from four reasons justifying ‘freedom of association', with the correct response 
being C (‘Having many organizations for people to join is important because this 
provides “opportunities to express different points of view’”). National percentages 
of correct responses range from 46 to 82 percent. With an international mean of 
69 percent correct, this item is clearly among the easier ones in the test. It calls 
for basic background knowledge, or perhaps some recall of politics-related 
experience, to rule out the incorrect responses and to select the correct one. 



Figure 3.1e Item Example: Function of having more than one political party 



Country 


Correct Answers 




Example 8 (Item #11) 




(in %) 




Type 1: Knowledge of Content 


Australia 


75 


(1.3) 






Belgium (French) 


67 


(1.7) 






Bulgaria 


70 


(1.6) 






Chile 


60 


(1.2) 






Colombia 


54 


(1.6) 






Cyprus 


88 


(0.9) 


11. 


In democratic countries what is 


Czech Republic 


79 


(1.0) 




the function of having more than 


Denmark 


84 


(0.8) 




one political party? 


England 


78 


(1.0) 


A. 


Estonia 


62 


(1.2) 


To represent different opinions 
[interests] in the national 


Finland 


80 


(1.0) 




Germany 


84 


(0.9) 




legislature Ce.g. Parliament, 


Greece 


85 


(0.7) 




Congress].* 


Hong Kong (SAR) 


76 


(1.1) 


B. 


To limit political corruption. 


Hungary 


75 


(1.2) 


C. 


To prevent political 
demonstrations. 


Italy 


86 


(0.9) 


D. 


Latvia 


57 


(1.7) 


To encourage economic 


Lithuania 


68 


(1.2) 




competition. 


Norway 


83 


(0.7) 






Poland 


82 


(1.1) 






Portugal 


84 


(0.8) 






Romania 


67 


(1.7) 






Russian Federation 


71 


(1.6) 






Slovak Republic 


77 


(1.0) 






Slovenia 


81 


(0.7) 






Sweden 


75 


(1.5) 






Switzerland 


82 


(0.9) 






United States 


72 


(1.5) 






International Sample 


75 


(0.2) 







( ) Standard errorsappear in parentheses. 
* Correct answer. 



Source: IEA Civic Education Study , Standard Population of 14-year-olds tested in 1999. 
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The fifth example (Figure 3. le; example 8 in Figure 3.2) is another Type 1 item, 
this time relating to institutions and practices in a democracy. In this item, four 
potential functions of a political system with more than one party were presented 
to the students who had to select the correct one (A: ‘to represent different 
opinions [interests] in the national legislature’). The task turned out to be quite 
easy, with an international average percentage correct of 75 and a range across 
countries of 54 to 88 percent. In terms of its cognitive demands, this item is quite 
similar to the previous example, that is, a certain amount of political background 
knowledge and/or politics-related experience is needed if the correct response is 
to be identified. Some might argue that option B (‘to limit political corruption’) is 
not entirely wrong. The students who generally did well on the test, however, 
shared the conviction that the function of parties to represent different opinions 
or interests refers to a more fundamental role and is the more appropriate choice. 

Three more sample items from the IEA test are given in Appendix A (Figures 
A.la-c). They are intended to illustrate further the kind of questions and 
cognitive demands presented in the questions. Table A. 1 in Appendix A lists the 
domain categories and short titles for all items contained in the test, along with 
the percentages of correct responses (international means). 

ESTIMATION OF ITEM DIFFICULTIES AND STUDENT 
ABILITIES 

One can always rank the items of a test according to the respective percentages of 
correct responses, the easiest ones being those with the highest percentage of 
correct answers and the most difficult ones those with the lowest. These 
percentages, however, are not the most informative measures of difficulty, since 
they do not take into account the ability levels of those who answered the item 
correctly. Similarly, ‘percent correct’ (or the sum of correct responses) is not a very 
good measure of student ability, because it assumes implicitly that all items are 
equally difficult. 

When certain assumptions are met, it is possible to apply modern scaling 
techniques to arrive at an interval scale onto which measures of both item 
difficulty and student ability can be projected at the same time. The most 
important of these conditions is that a student with an ability thus determined 
solves — at a defined or reasonable level of success — all of the items with difficulties up 
to his or her ability level and fails most of the harder items. Following the 
example of the Third International Mathematics and Science Study (TIMSS), we 
chose in the present study a probability of .65 to represent this defined level of 
success. Because the simultaneous estimation of student abilities and item 
difficulties is based on probability functions, such an approach is sometimes called 
‘probabilistic’. The more widely used term is ‘Item Response Theory’ or ‘IRT’ (see, 
for example, Hambleton, Swaminathan Be Rogers, 1991). We used one of its 
models (the so-called one-parameter model that produces maximum likelihood 
estimates) to scale the 38 cognitive test items, once we had established that the 
required underlying assumptions held empirically. Because this model leaves 
researchers free to choose the mean and the standard deviation for the metric to 
be used, we set the international mean of the scale for civic knowledge (and two 
sub-scales, to be discussed later) to 100, with a standard deviation of 20. 
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Figure 3.2 International Difficulty Map for Sample Items Relating to Civic 
Knowledge 

vwvww 



Example 1 



identify fact about taxes 
(Type 2: Item #38) 

Scale Value: 109 

International Mean Correct (in %): 49 



Example 3 



identify what makes a 
government non-democratic 
(Type 1: Item #17) 

Scale Value: 106 

International Mean Correct (in %): 53 



Example 5 



identify main message of 
cartoon about history books 
(Type 2: Item #36) 

Scale Value: 102 

International Mean Correct (in %): 57 



Example 7 



identify why organisations are 
important in democracy 
(Type 1: Item #07) 

Scale Value: 93 

International Mean Correct (in %): 69 



110 



100 



90 



Example 2 



identify an example of 
discrimination in pay equity 
(Type 2: Item #26) 

Scale Value: 108 

International Mean Correct (in %): 50 



Example 4 



identify result if large publisher 
buys many newspapers 
(Type 1: Item #18) 

Scale Value: 103 

International Mean Correct (in %): 57 



Example 6 



identify party which 
issued a leaflet 
(Type 2: Item #23) 

Scale Value: 97 

International Mean Correct (in %): 65 



Example 8 



identify function of having 
more than one political party 
(Type 1: Item #11) 

Scale Value: 88 

International Mean Correct (in %): 75 



lAAAAAAAAl 



NOTE: Each item was placed onto the International Civic Knowledge Scale. Items are shown at the point on 
the scale where students with that level of proficiency had a 65 percent probability of providing a correct 
response. 

Because percentages and scale values are rounded to the nearest whole number, some results may appear 
inconsistent. 

Source: IE A Civic Education Study, Standard Population of 14-year-olds tested in 1999. 
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The item difficulty estimates produced by using an IRT model allowed us to 
demonstrate the substantive meaning of the items in the context of the test as a 
whole. Figure 3.2 provides an ‘item difficulty map’ that illustrates the relationship 
between the item-specific performance levels and the international knowledge test 
score as defined. 

In this figure, the item examples (including those given in Appendix A) are placed 
on the scale — the gray bar in the middle — at the point where a student with the 
respective ability has a probability of .65 to provide the correct answer. For 
example, a student with a scale score of at least 106 on the civic knowledge scale 
has a chance of two in three or better to identify correctly what makes a 
government non-democratic (item example 3). Similarly, students with scale scores 
lower than 97 will have less than a 65 percent chance to identify correctly the 
origin of the imaginary electoral leaflet (item example 6). 1 

We had determined that the probability level of .65, which links item difficulties 
and student abilities, should be set as a formal characteristic of the metric used. 
Substantively, however, we considered this to be an appropriate interpretation of 
the somewhat vague term ‘reasonable level of success’. It reflects our attempt to 
take into account knowledge expectations with a higher than 50 percent chance, 
even though this level is still below that which some might call mastery of the 
subject. 

Figure 3.2 also displays, for each item example, the international average of 
correct answers. For the test as a whole, including those items not illustrated here, 
the percentages of correct responses range from 35 percent for the hardest item in 
the test to 85 percent for the easiest one. The respective difficulty parameters, or 
scale values, are 77 for the easiest item and 121 for the hardest. A fairly wide 
range of difficulties and student abilities therefore could be covered by the test. 
Table A. 1 in Appendix A also contains the difficulty parameters for each item. 
Thus, it is possible to see how item difficulties are distributed across content 
domains. 




One crucial point in the selection of items was to ensure that the item difficulties, 
and consequently the estimated student abilities, were truly comparable across 
countries. We had anticipated that this assumption would be difficult to meet, that 
is, ‘differential item-functioning’ or ‘item-by-country interaction’ would occur. 
This is the case when an item of a certain international difficulty level is relatively 
easier or harder for students in a country than one would expect on the basis of 
that country’s overall mean. Different civic education curricula or differences in 
the historical and political context might cause such deviations in a country’s 
response pattern from the international findings. Although there were 1,064 item- 
by-country pairs (38 items for each of 28 countries), only eight of these showed a 
clear item-by-country interaction. Because there were so few, we decided not to 
re-estimate (‘float’) the item difficulty parameters for the countries concerned, but 
to rely on the fact that the interaction effects within a country sum to zero. 



Another potential problem was that, in a particular country, some items might not 
discriminate well between high- and low-achieving students. This happens when 
many able students in that country fail to choose the correct response, or when 
many weaker students do choose it (‘item misfit’). Again, we found that such 
deviations from the international test characteristics were rare: we encountered 33 
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instances out of the 1,064. To ensure full international comparability, however, we 
deemed it appropriate to exclude the misfitting items countrywise from the final 
analysis of scores by treating them statistically as not having been administered. 
We therefore re-estimated all parameters, taking these cases into account. In no 
case is a country's civic knowledge score based on fewer than 35 out of the 38 
available items, and in no case did we implicitly penalize a country for not being 
scored on the full set of 38 items. For further detail, see the technical report 
(Lehmann etaL, forthcoming). 

In general, careful analysis showed us that it was possible to construct a 
meaningful, reliable and valid international test of student knowledge about 
democratic institutions, principles, processes and related topics that has a high 
degree of comparability across countries. 

CIVIC KNOWLEDGE ACROSS COUNTRIES 

Figure 3.3 presents a summary of the results of the international test on civic 
knowledge for all participating countries. To aid interpretation of these results, 
we added the date of testing, the tested grade and the average age of students to 
the figure. For some countries, readers should also take into account the specific 
information contained in the footnotes and the fuller elaboration given in 
Chapter 2. 

On the basis of the scaling technique just explained, the table within Figure 3.3 
contains the average test score and the standard error of sampling for each 
participating country. The international mean of the distribution (based on the 28 
equally weighted national samples) and its standard error are included to give an 
orientation mark for each country to compare itself to the international average. 
Thus, the table indicates which countries differ significantly (after correction for 
multiple comparisons) from the international average. 

Ten countries have means that are significantly higher and eight countries have 
means that are significantly lower than the international mean. The remaining ten 
countries belong to a middle group with country means that do not differ 
significantly from the international mean. For the most part, differences between 
countries within the three major groups are not significant (Figure 3.4). 

Poland, the Czech Republic and the Slovak Republic, three post-communist 
countries from Central and Eastern Europe, belong to the top group. Other 
countries in this group are the Greek-speaking countries Greece and Cyprus, the 
United States, Italy and two Nordic countries (Finland and Norway). Students 
from Hong Kong (SAR) also perform significantly better than the international 
average. 

In the group of countries with means significantly below average, only Portugal 
and French-speaking Belgium are in Western Europe, and both tested very young 
students. Romania and the three Baltic countries Estonia, Latvia and Lithuania 
have average scores between 92 and 94. Chile and Colombia, the two Latin 
American countries in this study, have the lowest means (88 and 86 respectively). 
Here, 75 percent of the students have scores below the international average. 
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Figure 3.3 Distributions of Civic Knowledge 
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( ) Standard errors appear in parentheses. 






| Percentiles of Performance 1 




▲ Country mean significantly higher than international mean. 
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▼ Country mean significantly lower than international mean. 






Mean ana t/unnaence interval 
(±2SE) 


1 Countries with testing date at beginning of school year. 










2 National Desired Population does not cover all International Desired Population. 






3 Countries did not meet age/grade specification. 










4 Countries' overall participation rate after replacement less than 85 percent. 






5 In Portugal, Grade 8 selected instead of Grade 9 due to average age. Mean scale score for Grade 9 was 106. 


* In Switzerland, Grade 8 was tested mainly in German cantons; Grade 9 mainly in French and Italian cantons. In 


the Russian Federation, students in Grade 9 have eight or nine years of schooling depending on the duration of the 


primary school they finished. In 1999 about 70 percent of Russian students tested had eight years of schooling at 


the end of Grade 9. 












** Information on age was not available for Germany. International mean age based on 27 countries only. 





Source: IEA Civic Education Study, Standard Population of 14-year-olds tested in 1999. 
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A closer look at these averages reveals the necessity to consider mean age 
differences. At the country level, the national average of civic knowledge is 
correlated with the mean age of the sample (r= .37). Thus, countries with an 
older sample have an advantage over those that tested younger students. While it 
would be possible to adjust national averages of knowledge for differences in 
mean age (on the assumption of equal growth with age across all countries), such 
adjustment changes relatively little in the rank order of countries: the correlation 
between adjusted and unadjusted country means is r = ,92. 2 The issue of age and 
growth will be explored more fully in a future report. 

It is beyond the scope of the present volume to try and explain these country 
differences. Apart from the mean age of the sample, however, other factors can be 
shown to be correlated, such as the economic situation (gross national product per 
capita, r= .32, or the general level of literacy, r = .26; see Tables 1.1 and 1.2). 

But again, these findings should be taken primarily as evidence that more 
thorough analyses are needed, both at the international level and the country level 
with respect to individual and classroom-level processes. 

The graphic representation on the right-hand side of Figure 3.3 shows the 
distribution of civic knowledge in the tested grade for each country. The 
cognitive score is shown for the mean as well as for the 5th, 25th, 75th and 95th 
percentiles. Each percentile point indicates the percentage performing below and 
up to the respective score. To give an example, 25 percent of the students perform 
below and up to the score marked by the 25th percentile. Seventy-five percent 
have attained a result above the corresponding score. The dark boxes in the center 
of each distribution stand for the country means and their 95 percent confidence 
interval (mean ± two standard errors of sampling). For an approach that gives 
substantive meaning to these percentiles, see Panel 3.2. 

The variation of means between countries gives further information about the 
characteristics of this subject area. Twenty-five of the 28 countries differ by less 
than half of a standard deviation from the international average. Exceptions are 
Poland, with the highest national average score of 1 1 1, and Chile and Colombia, 
with mean scores below 90. This range can be compared to the results of other 
international studies on educational achievement. First, it is similar to that of the 
IEA Reading Literacy Study (Elley, 1994). Fifteen of the 28 countries 
participating in the Civic Education Study also participated in that study. The 
range between the highest and the lowest ranking country was .79 of an 
international standard deviation, compared with .70 in the present study. 

Secondly, 22 of the 28 countries also participated in the Third International 
Mathematics and Science Study (Beaton, Mullis, Martin, Gonzalez, Kelly & Smith, 
1996). Among these countries, country means for mathematics achievement 
differed by 1.79 standard deviations between the highest and the lowest 
achieving country (and still by 1.10 standard deviations if the special case of 
Colombia was excluded). One of the reasons for this difference between the 
Reading Literacy Study and the Civic Education Study on the one hand and 
TIMSS on the other may be that reading (at the age of 14) and civic knowledge 
are less closely linked to curriculum and instruction than is mathematics. 
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PANEL 3.2 How to Interpret the Results for the International Distribution of 
Civic Knowledge 



It is informative to attribute substantive meaning to characteristics of the distribution, such as 
means and percentiles, by relating them to the item-difficulty parameters. These are indicated in 
the item examples given and in the item overview presented in Table A. I, Appendix A. This 
procedure can be done for each country separately. The following provides a demonstration of 
this mode of interpretation for the results for the weighted international file. 

The 5th percentile in the international sample corresponds to a scale value of 71, which is lower 
than the item difficulty parameter of the easiest item in the test. This is Item 16, scale value 77, 
which pertains to the domain of national identity and international relations and which asks 
students to identify the major purpose of the United Nations (Table A.l, Appendix A). Thus, the 
probability that the lowest achieving students — internationally speaking — will respond correctly 
to this task is less than .65. In contrast, the probability that some students among the lowest 5 
percent in a number of countries (for example, Cyprus, the Slovak Republic and Finland) will be 
able to identify the main purpose of the United Nations at the defined ‘reasonable level of 
success’ is .65 or higher. 

Similarly, the 25th percentile in the international score distribution corresponds to a scale value 
of 85. This value is close to the item difficulty parameter (scale value 88) of the example given in 
the text (Figure 3.1e) in which students were asked to identify the function of having more than 
one political party in a democratic country. More than 7 5 percent of the students in the 
participating countries have a probability of .65 or higher of being able to respond correctly to 
this item, which refers to one of the fundamental traits of representative democracies. Twenty- 
five percent of the students in the international sample were found to lack such a level of civic 
knowledge as exemplified by this item (and others of equal difficulty). Another item of 
approximately this difficulty level is, for instance, Item 12, which asks who ought to govern in a 
democracy (Appendix Table A.l). 

The international mean of 100 corresponds to an item also pertaining to the domain of 
democracy and its defining characteristics. This is Item 14, which calls on students to evaluate 
the strengths and weaknesses of democratic systems by identifying the main message of a 
cartoon about differences of opinion in a democracy. The typical student across participating 
countries has a reasonable chance (about two in three) to respond correctly to tasks of this 
difficulty level (Appendix A, Table A.l). 

The 75th percentile, separating the top quarter from the lower 75 percent in the distribution, 
was found at a scale value of 1 12. This difficulty level exceeds the difficulty parameters of all of 
the examples given above. Item 22, which asks about the functions of periodic elections and 
which also was used by IEA in 1971, comes closest, with a difficulty level of 1 13. Thus, a little 
less than one out of four students in the participating countries was — with the defined minimal 
degree of likelihood — able to respond correctly to this item or others of the same level of 
difficulty. In some countries, most notably in Poland, the national mean is almost at the level of 
the international 75th percentile. 

The 95th percentile, finally, indicates the lower bound of achievement for the top 5 percent in 
the international distribution of civic knowledge. It is found at a scale value of 135, which 
implies that in this international top group the probability of success is higher than .65 for all 
the items in the test. 3 This threshold is attained by the highest achieving students in a good 




CHAPTER 3 CIVIC KNOWLEDGE 



5 a 



57 




Panel 3.2 continued 

number of participating countries (given here in the order of national means): the Slovak 
Republic, the Czech Republic, Australia, Hungary, Slovenia, Denmark, Germany, England, 
Sweden and Bulgaria. Another group of countries where the top 5 percent of the students were 
even more successful on the test includes Poland, Finland, Cyprus, Greece, Hong Kong (SAR), 
the United States, Italy, Norway and the Russian Federation. 

The diagram in Figure 3.3 thus provides considerably more information than a simple ranking 
of countries by national average. The distributional properties for each country can and should 
be studied, preferably in close linkage with the appropriate statistics (such as the percentiles 
that were used here) and the item characteristics of the test. 



Figure 3.4 provides a visual presentation of country averages showing those that 
differ significantly from each other at the 95 percent significance level. By 
selecting a country and reading across the table, we can see that the triangles 
pointing up indicate a significantly higher average performance than the country 
listed across the top, whereas the triangles pointing down stand for a significantly 
lower average. Dots indicate that the two country averages do not differ 
significantly from each other. 4 

The differences upon which this multiple-comparison table is based are mostly 
quite small between adjacent countries. For example, the country mean in Poland, 
where students have the highest average of all participating countries, does not 
significantly differ from the next seven countries. The Colombian average does 
not differ significantly from the Chilean one, but it does so from all other country 
means. Given that sampling errors are not identical across countries, it is possible 
that in some cases an apparently smaller difference is statistically significant while 
a larger one is not. 

There is no obvious pattern in the distribution of national mean scores on the 
civic knowledge scale. Western European, North American and post-Communist 
countries lie in the top group of countries. The two Latin American countries and 
the Baltic countries as well as Romania have similar country means on the IEA 
civic knowledge test. We have already stated that many factors are likely to be 
involved in the emergence of between-country differences, even at the level of 
aggregate national indicators. 

DISTINGUISHING BETWEEN CONTENT KNOWLEDGE AND 
INTERPRETATIVE SKILLS 

As was shown above, the IEA test was designed to contain two different types of 
items: Type 1, tapping the students’ knowledge of content; and Type 2, 
measuring their skills in interpreting civic-related material (cartoons, leaflets, 
descriptions of issues) and also incorporating their ability to distinguish between 
facts and opinions. It is also clear from the previous analysis that a one- 
dimensional representation of the students’ response patterns is a psychometrically 
sound and meaningful way to present the findings. In this analysis each student is 
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Figure 3.4 Multiple Comparisons of Civic Knowledge 

Instructions: Read across the row for a country to compare performance with the countries listed 
in the heading of the chart. The symbols indicate whether the mean achievement of the country 
in the row is significantly lower than that of the comparison country, significantly higher than 
that of the comparison country, or if there is no statistically significant difference between the 
two countries. 
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Countries are ordered by mean achievement across the heading and down the rows. 



▲ Mean achievement significantly higher than comparison country. 

• No statistically significant difference from comparison country. 

▼ Mean achievement significantly lower than comparison country. 

NOTE: Significance tests at .05 level, adjusted for multiple comparisons. 

1 Countries with testing date at beginning of school year. 

2 National Desired Population does not cover all International Desired Population. 

3 Countries did not meet age/grade specification. 

4 Countries' overall participation rate after replacement less than 85 percent. 

5 In Portugal, Grade 8 selected instead of Grade 9 due to average age. Mean scale score for Grade 9 was 106. 
Source: IE A Civic Education Study, Standard Population of 14-year-olds tested in 1999. 
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simply awarded one total civic knowledge score, and from these scores the 
reported national means and distributions presented in the last section are derived. 

Even though this method is completely defensible, it was of interest to investigate 
whether the distinction between the two item types would map the students’ 
response patterns even more appropriately. Might it be possible to derive a sub- 
score for content knowledge and for skills in interpreting civic-related material 
that would allow additional insights? To ascertain the dimensional structure of the 
IEA test under the assumption that two dimensions (corresponding to content 
knowledge and interpretative skills) could be distinguished, we performed 
confirmatory factor analyses (CFA). These were based on a calibration sample of 
500 randomly selected students per country. As a contrast to the one-dimensional 
model with all 38 items on a single factor, we estimated a second model that 
allocated the 25 Type 1 items to one factor (content knowledge) and the 
remaining 13 items to a second factor (interpretative skills). 5 

Figure 3.5 shows a graphical display of the two-factor-structure and the results of 
a comparison of the model fit with the one-factor-solution. The diagram includes 
the factor correlation between the two latent constructs ‘knowledge of content’ 
and ‘skills of interpretation’, as well as the factor loading for each item and the 
proportion of (unexplained) error. The variance of the items explained by the 
latent dimensions ranges from 17 to 53 percent. 

Both models show good statistical properties, but the two-factor-solution that is 
shown graphically has a relatively better model fit than the one-factor-solution. 
The difference in the chi-square statistic is 604, with one degree of freedom, 
which is highly significant. 6 However, the two factors are strongly correlated 
with each other (r = .91), which indicates that the two abilities — ‘content 
knowledge’ and ‘interpretative skills’ — refer to highly similar but not identical 
aspects of student performance. While it is true that the students who know much 
about civic-related content are likely also to have highly developed skills in 
interpreting civic-related material, this is not always the case. It is also possible 
that systematic differences occur between groups of students and even countries. 
Based on these results, we decided to present, in addition to the total cognitive 
score on civic knowledge, findings based on the two component sub-scales content 
knowledge and skills in interpreting civic-related information. 

In Figure 3.6, we have again ranked the participating countries in the order of 
average achievement in civic knowledge (total score), but we also have given the 
means for the sub-scales of content knowledge and interpretative skills. All three 
scales are one-parameter Rasch scales, and all three have been set to an 
international mean of 100 and an international standard deviation of 20. It is not 
our assumption that one of the sub-scales refers to more complex or superior 
abilities than the other. On the contrary: in recognition of the fact that scale 
properties always depend on the particular choice of items used, our underlying 
assumption is that there is no meaningful way of comparing directly the scores on 
one of the sub-scales with those on the other. Given, however, that the students 
who took the items from the two sub-scales were identical, it is meaningful to 
compare the performance of countries on the two sub-scales, relative to the respective 
international means. In a sense, this analysis amounts to investigating the differential 
functioning of two groups of items that can be considered as measuring two 
different aspects of civic knowledge. 
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Figure 3.5 Confirmatory Factor Analysis of Civic Knowledge Items 
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Comparison of Model Fit for One- and Two-Factor-Solution * 
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AGFI 
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Chi-Square / DF 


One-Factor-Solution 


0.045 
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Two-Factor-Solution 


0.041 
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0.91 


0.90 
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Model Comparison 



604/1 



* Models estimated with Weighted Least Square (WLS) and tetrachoric correlation matrices. 
Chi-square values corrected for non-normality. Calibration sample with 500 students per 
country. 



Source: IEA Civic Education Study, Standard Population of 14-year-olds tested in 1999. 
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Figure 3.6 demonstrates that the outcomes on the two sub-scales do vary 
somewhat within countries. In 26 countries (the exceptions are Denmark and 
Germany), either the content or skills sub-score average is significantly above or 
below the international mean. In contrast, the total civic knowledge score reveals 
that 18 of the countries are above or below the international mean. This suggests 
that looking at two sub-scores rather than a single civic knowledge score does 
contribute to an understanding of relative strengths and weaknesses of civic 
knowledge as developed in the participating countries. 

In five countries — Australia, England, Sweden, Switzerland and the United 
States — relative to the sub-scale means, the items pertaining to the skills 
dimension are more likely to be answered correctly than those from the content 
dimension. In two countries, Poland and the Russian Federation, relative 
performance on the content items is better than that on the skills items. 

Once again these findings depend heavily on the specific items in the respective 
sub-scales and therefore can only be interpreted as pertaining to the relative 
position of countries on the two sub-scales. Nevertheless, it is interesting that, in 
some countries, students have acquired their content knowledge and interpretative 
skills differentially. In the absence of any obvious explanation, it is all the more 
important to systematically investigate such patterns in future analyses and to 
relate results to the findings from Phase 1 (Torney-Purta, Schwille & Amadeo, 
1999; Steiner-Khamsi, Torney-Purta & Schwille, forthcoming). 

GENDER DIFFERENCES IN CIVIC KNOWLEDGE 

Most previous research has shown gender differences regarding cognitive 
performance of students in this area. In particular, according to studies conducted 
a decade or more ago, males scored higher on civic knowledge tests than females. 
In the first IEA Civic Education Study (1971), gender had a significant effect on 
cognitive achievement among the 14-year-old students in four out of eight 
countries, and these gender differences became more notable among older 
students, with males consistently outscoring females (Torney etal, 1975, p.148). 
National studies mostly have had similar results on different tests measuring 
political knowledge (see Panel 3.1). 

Figure 3.7 shows the gender differences in civic knowledge for all 28 countries in 
the present study. After correcting for multiple comparisons, we found no 
statistically significant differences in 27 of the 28 countries. 7 The exception is 
Slovenia, where females perform better than males. 

These findings suggest that, at least in a simple comparison, among 14-year-olds 
in most countries, political content knowledge and skills in interpreting political 
communication are unrelated to gender and that the previously found dominance 
of males in this area, even at the age of 14, is greatly diminished. This 
interpretation needs some modification when other variables are taken into 
account, however. More about this topic will be said in Chapter 8. 
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Figure 3.6 Content Knowledge Subscore and Interpretative Skills Subscore 
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( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may 



appear inconsistent. 



▲ Country mean significantly higher than international mean. 

• No statistically significant difference between country mean and 
international mean. 

▼ Country mean significantly lower than international mean. 



NH = Mean Subscore Civic Content (± 2 SE). 
h#-| = Mean Subscore Civic Skills (± 2 SE). 

D = Mean Total Civic Knowledge Score. 



1 Countries with testing date at beginning of school year. 

2 National Desired Population does not cover all International Desired Population. 

3 Countries did not meet age/grade specification. 

4 Countries' overall participation rate after replacement less than 85 percent. 

5 In Portugal, Grade 8 selected instead of Grade 9 due to average age. Mean scores for Grade 9 were 108 on the Civic 
Content Scale, 103 on the Civic Skills Scale and 106 on the Total Civic Knowledge Scale. 



Source: IEA Civic Education Study , Standard Population of 14-year-olds tested in 1999. 
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